[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:48:59 dist-test-slave-z2s5 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:48:59 dist-test-slave-z2s5 krb5kdc[399](info): set up 2 sockets
Sep 25 14:48:59 dist-test-slave-z2s5 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:49:00.190864 392 mini_kdc.cc:121] Time spent starting KDC: real 1.126s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 14:49:00.468053 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.277s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:00 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:00.568969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:00.651993 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:00.652258 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:00.652297 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:00.653573 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:00.653641 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:00.653664 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:00.653694 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:00.653726 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:00.655315 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:00.655609 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:00.655896 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:00.661713 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:00.661738 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:00.661801 412 server_base.cc:1034] running on GCE node
W20260925 14:49:00.661721 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:00.662200 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:00.662578 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:00.663755 412 hybrid_clock.cc:648] HybridClock initialized: now 1790347740663702 us; error 77 us; skew 500 ppm
Sep 25 14:49:00 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:00.666450 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:00.667057 412 webserver.cc:460] Webserver started at http://127.0.98.62:42677/ using document root <none> and password file <none>
I20260925 14:49:00.667335 412 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:00.667411 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:00.667569 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:00.668676 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "2f6cca6f22594b76a10dbc3542fdfb0d"
format_stamp: "Formatted at 2026-09-25 14:49:00 on dist-test-slave-z2s5"
server_key: "36ee3f27e93c8e338681d952307407b1"
server_key_iv: "fcea551419318290ead6d1723ad98699"
server_key_version: "encryptionkey@0"
I20260925 14:49:00.669083 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f6cca6f22594b76a10dbc3542fdfb0d"
format_stamp: "Formatted at 2026-09-25 14:49:00 on dist-test-slave-z2s5"
server_key: "36ee3f27e93c8e338681d952307407b1"
server_key_iv: "fcea551419318290ead6d1723ad98699"
server_key_version: "encryptionkey@0"
I20260925 14:49:00.670585 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:00.671445 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:00.671619 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:00.671725 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "2f6cca6f22594b76a10dbc3542fdfb0d"
format_stamp: "Formatted at 2026-09-25 14:49:00 on dist-test-slave-z2s5"
server_key: "36ee3f27e93c8e338681d952307407b1"
server_key_iv: "fcea551419318290ead6d1723ad98699"
server_key_version: "encryptionkey@0"
I20260925 14:49:00.671860 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:00.697026 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:00.699261 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:00.699463 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:00.703313 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46863
I20260925 14:49:00.703364 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46863 every 8 connection(s)
I20260925 14:49:00.703775 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:00.704474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 412
I20260925 14:49:00.704600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:00.705109 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:00.705488 392 external_mini_cluster.cc:1428] Setting key 1cc4150dc316a419acabf3781a5e2d9b
I20260925 14:49:00.708256 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d: Bootstrap starting.
I20260925 14:49:00.709007 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:00.709363 480 log.cc:824] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:00.710263 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d: No bootstrap required, opened a new log
I20260925 14:49:00.712888 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260925 14:49:00.713093 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:00.713136 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f6cca6f22594b76a10dbc3542fdfb0d, State: Initialized, Role: FOLLOWER
I20260925 14:49:00.713291 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260925 14:49:00.713380 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:00.713419 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:00.713472 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:00.714252 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260925 14:49:00.714370 480 leader_election.cc:302] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f6cca6f22594b76a10dbc3542fdfb0d; no voters:
I20260925 14:49:00.714581 480 leader_election.cc:288] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:00.714634 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:00.714816 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [term 1 LEADER]: Becoming Leader. State: Replica: 2f6cca6f22594b76a10dbc3542fdfb0d, State: Running, Role: LEADER
I20260925 14:49:00.714860 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:00.714907 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } }
I20260925 14:49:00.715201 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260925 14:49:00.715305 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:00.715476 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f6cca6f22594b76a10dbc3542fdfb0d. Latest consensus state: current_term: 1 leader_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f6cca6f22594b76a10dbc3542fdfb0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46863 } } }
I20260925 14:49:00.715561 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:00.715672 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:00.716255 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:00.719174 502 mvcc.cc:205] Tried to move back new op lower bound from 7333264345978372096 to 7333264345968480256. Current Snapshot: MvccSnapshot[applied={T|T < 7333264345978372096}]
Sep 25 14:49:00 dist-test-slave-z2s5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347740, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:00.719596 491 catalog_manager.cc:1347] Generated new cluster ID: fcb03f08c1d04d3295e49a0cb3bddff0
I20260925 14:49:00.719684 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:00.723893 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:00.706584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46448 (local address 127.0.98.62:46863)
0925 14:49:00.706796 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:00.706806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:00.706981 (+ 175us) server_negotiation.cc:407] Connection header received
0925 14:49:00.709301 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:00.709312 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:00.709380 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:00.709549 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:00.712035 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:00.712936 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:00.714716 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:00.715393 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:00.715543 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:00.720052 (+ 4509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:00.720056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:00.720056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:00.720068 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:00.721817 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:00.722256 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:00.722259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:00.722259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:00.722347 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:00.722643 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:00.722644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:00.722644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:00.722885 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:00.723470 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:00.723560 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:49:00.723702 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260925 14:49:00.729979 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:00.730571 491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:00.737054 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f6cca6f22594b76a10dbc3542fdfb0d: Generated new TSK 0
I20260925 14:49:00.737268 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:00.765494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:00.847858 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:00.848104 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:00.848145 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:00.849310 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:00.849381 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:00.849427 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:00.850991 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:00.851262 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:00.851496 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:00.856746 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:00.856752 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:00.857067 506 server_base.cc:1034] running on GCE node
W20260925 14:49:00.856752 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:00.857366 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:00.857648 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:00.858808 506 hybrid_clock.cc:648] HybridClock initialized: now 1790347740858791 us; error 34 us; skew 500 ppm
Sep 25 14:49:00 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:00.861945 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:00.862498 506 webserver.cc:460] Webserver started at http://127.0.98.1:43305/ using document root <none> and password file <none>
I20260925 14:49:00.862751 506 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:00.862818 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:00.862968 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:00.864092 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f8cad2ebb24435788aaf5132159127e"
format_stamp: "Formatted at 2026-09-25 14:49:00 on dist-test-slave-z2s5"
server_key: "a85035eb0bbbae47f976c605206dce33"
server_key_iv: "72b9bb4c0da4f15606eb8a48a2963630"
server_key_version: "encryptionkey@0"
I20260925 14:49:00.864491 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f8cad2ebb24435788aaf5132159127e"
format_stamp: "Formatted at 2026-09-25 14:49:00 on dist-test-slave-z2s5"
server_key: "a85035eb0bbbae47f976c605206dce33"
server_key_iv: "72b9bb4c0da4f15606eb8a48a2963630"
server_key_version: "encryptionkey@0"
I20260925 14:49:00.866016 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:00.867100 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:00.867349 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:49:00.867466 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "1f8cad2ebb24435788aaf5132159127e"
format_stamp: "Formatted at 2026-09-25 14:49:00 on dist-test-slave-z2s5"
server_key: "a85035eb0bbbae47f976c605206dce33"
server_key_iv: "72b9bb4c0da4f15606eb8a48a2963630"
server_key_version: "encryptionkey@0"
I20260925 14:49:00.867558 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:00.875866 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:00.878151 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:00.878284 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:00.878551 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:00.878837 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:00.878871 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:00.878896 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:00.878911 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:00.885190 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34491
I20260925 14:49:00.885253 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34491 every 8 connection(s)
I20260925 14:49:00.885596 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:00.890702 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 506
I20260925 14:49:00.890803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:00.890986 392 external_mini_cluster.cc:1428] Setting key 827a1fc12191846dd35cec2f0a47e419
Sep 25 14:49:00 dist-test-slave-z2s5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:00.896386 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:00.886511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58101 (local address 127.0.98.62:46863)
0925 14:49:00.886601 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:00.886604 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:00.886835 (+ 231us) server_negotiation.cc:407] Connection header received
0925 14:49:00.887349 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:00.887350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:00.887363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:00.887414 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:00.888418 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:00.888937 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:00.889953 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:00.890181 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:00.890270 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:00.892948 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:00.892950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:00.892951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:00.892960 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:00.894693 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:00.895169 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:00.895171 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:00.895171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:00.895241 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:00.895559 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:00.895560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:00.895561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:00.895724 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:00.895802 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:00.896132 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 14:49:00.896236 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:49:00.896422 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:00.886452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.1:58101)
0925 14:49:00.886665 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:00.886702 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:00.887218 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:00.887502 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:00.887506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:00.888088 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:00.888307 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:00.888312 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:00.889040 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:00.889041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:00.889859 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:00.889862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:00.890279 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:00.890286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:00.890467 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:00.891095 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:00.891106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:00.892831 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:00.894809 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:00.894816 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:00.894818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:00.895078 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:00.895350 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:00.895352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:00.895352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:00.895480 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:00.895828 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:00.895835 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:00.895974 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:49:00.896077 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:49:00.896225 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260925 14:49:00.897235 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260925 14:49:00.897359 635 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:00.897544 635 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260925 14:49:00.898123 444 ts_manager.cc:194] Registered new tserver with Master: 1f8cad2ebb24435788aaf5132159127e (127.0.98.1:34491)
I20260925 14:49:00.898845 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58101
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:00.918313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:00.999747 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:00.999939 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:00.999961 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:01.000980 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:01.001013 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:01.001037 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:01.002377 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:01.002668 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:01.002912 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:01.008003 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:01.008078 640 server_base.cc:1034] running on GCE node
W20260925 14:49:01.008020 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:01.008167 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:01.008419 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:01.008611 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:01.009764 640 hybrid_clock.cc:648] HybridClock initialized: now 1790347741009749 us; error 39 us; skew 500 ppm
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:01.012277 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:01.012701 640 webserver.cc:460] Webserver started at http://127.0.98.2:41221/ using document root <none> and password file <none>
I20260925 14:49:01.012873 640 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:01.012912 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:01.012997 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:01.013942 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "634ac615a8da493a9c36c018f5e252b1"
format_stamp: "Formatted at 2026-09-25 14:49:01 on dist-test-slave-z2s5"
server_key: "14be731e4c0da9dea0528b7ace0c20e2"
server_key_iv: "8d2a9ffdfec1dd7bbc8c3c39d7c65f15"
server_key_version: "encryptionkey@0"
I20260925 14:49:01.014245 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "634ac615a8da493a9c36c018f5e252b1"
format_stamp: "Formatted at 2026-09-25 14:49:01 on dist-test-slave-z2s5"
server_key: "14be731e4c0da9dea0528b7ace0c20e2"
server_key_iv: "8d2a9ffdfec1dd7bbc8c3c39d7c65f15"
server_key_version: "encryptionkey@0"
I20260925 14:49:01.015760 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:01.016567 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:01.016786 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:01.016887 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "634ac615a8da493a9c36c018f5e252b1"
format_stamp: "Formatted at 2026-09-25 14:49:01 on dist-test-slave-z2s5"
server_key: "14be731e4c0da9dea0528b7ace0c20e2"
server_key_iv: "8d2a9ffdfec1dd7bbc8c3c39d7c65f15"
server_key_version: "encryptionkey@0"
I20260925 14:49:01.016968 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:01.035256 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:01.037071 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:01.037235 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:01.037470 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:01.037775 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:01.037808 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:01.037830 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:01.037843 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:01.044703 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41587
I20260925 14:49:01.044762 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41587 every 8 connection(s)
I20260925 14:49:01.045109 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:01.053292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 640
I20260925 14:49:01.053395 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:01.053588 392 external_mini_cluster.cc:1428] Setting key 3e945934662783f48a78a150e4260ac8
I20260925 14:49:01.054165 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:01.046000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37427 (local address 127.0.98.62:46863)
0925 14:49:01.046105 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:01.046107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:01.046120 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:49:01.046602 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:01.046603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:01.046613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:01.046676 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:01.047602 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.048125 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.049064 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.049251 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.049306 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:01.051461 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:01.051462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:01.051463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:01.051469 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:01.052659 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.053037 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.053038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.053038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.053084 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.053414 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.053416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.053416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.053562 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:01.053623 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:01.053984 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:49:01.054075 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:01.054276 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:01.045839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.2:37427)
0925 14:49:01.046002 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:01.046044 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:01.046519 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:01.046742 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:01.046746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:01.047328 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:01.047514 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:01.047518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.048206 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.048206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:01.048991 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:01.048993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.049323 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.049324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:01.049417 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:01.049967 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:01.049976 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:01.051377 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:01.052757 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:01.052762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:01.052764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:01.052986 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:01.053151 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:01.053152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:01.053152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:01.053299 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:01.053634 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:01.053640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:01.053803 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 14:49:01.053930 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:49:01.054069 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260925 14:49:01.054826 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260925 14:49:01.054904 769 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:01.055092 769 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260925 14:49:01.055444 444 ts_manager.cc:194] Registered new tserver with Master: 634ac615a8da493a9c36c018f5e252b1 (127.0.98.2:41587)
I20260925 14:49:01.055907 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37427
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:01.077549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:01.162305 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:01.162647 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:01.162704 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:01.164024 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:01.164101 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:01.164160 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:01.165817 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46863
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:01.166108 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:01.166359 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:01.171716 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:01.171689 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:01.171710 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:01.171744 774 server_base.cc:1034] running on GCE node
I20260925 14:49:01.172029 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:01.172272 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:01.173437 774 hybrid_clock.cc:648] HybridClock initialized: now 1790347741173422 us; error 28 us; skew 500 ppm
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:01.176040 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:01.176419 774 webserver.cc:460] Webserver started at http://127.0.98.3:37983/ using document root <none> and password file <none>
I20260925 14:49:01.176652 774 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:01.176697 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:01.176806 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:01.177826 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "721ea7baa5104b749100a45ac19c38c2"
format_stamp: "Formatted at 2026-09-25 14:49:01 on dist-test-slave-z2s5"
server_key: "bac5fe3982eaa674a2517354fc407a26"
server_key_iv: "26b46e66a7e2048dd01c076ca4bc7e95"
server_key_version: "encryptionkey@0"
I20260925 14:49:01.178181 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "721ea7baa5104b749100a45ac19c38c2"
format_stamp: "Formatted at 2026-09-25 14:49:01 on dist-test-slave-z2s5"
server_key: "bac5fe3982eaa674a2517354fc407a26"
server_key_iv: "26b46e66a7e2048dd01c076ca4bc7e95"
server_key_version: "encryptionkey@0"
I20260925 14:49:01.179584 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:01.180361 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:01.180541 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:01.180637 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "721ea7baa5104b749100a45ac19c38c2"
format_stamp: "Formatted at 2026-09-25 14:49:01 on dist-test-slave-z2s5"
server_key: "bac5fe3982eaa674a2517354fc407a26"
server_key_iv: "26b46e66a7e2048dd01c076ca4bc7e95"
server_key_version: "encryptionkey@0"
I20260925 14:49:01.180718 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:01.191493 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:01.193293 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:01.193459 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:01.193671 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:01.193989 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:01.194047 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:01.194088 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:01.194116 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:01.200730 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42953
I20260925 14:49:01.200811 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42953 every 8 connection(s)
I20260925 14:49:01.201124 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:01.202126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 774
I20260925 14:49:01.202235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:01.202466 392 external_mini_cluster.cc:1428] Setting key 90efd413a8c08c5e887b597ed66a500c
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:01.210637 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:01.201884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52651 (local address 127.0.98.62:46863)
0925 14:49:01.201930 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:01.201932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:01.202102 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:49:01.202767 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:01.202769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:01.202780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:01.202835 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:01.203782 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.204278 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.205299 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.205499 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.205583 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:01.208065 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:01.208066 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:01.208066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:01.208073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:01.209244 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.209651 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.209652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.209652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.209709 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.210027 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.210028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.210028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.210146 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:01.210227 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:01.210500 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 14:49:01.210575 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:01.210803 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:01.201830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46863 (local address 127.0.98.3:52651)
0925 14:49:01.201980 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:01.202032 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:01.202667 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:01.202908 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:01.202912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:01.203477 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:01.203660 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:01.203665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.204391 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.204392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:01.205208 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:01.205209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.205590 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.205591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:01.205717 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:01.206175 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:01.206182 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:01.207973 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:01.209342 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:01.209348 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:01.209350 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:01.209561 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:01.209775 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:01.209776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:01.209776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:01.209914 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:01.210228 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:01.210232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:01.210338 (+ 106us) client_negotiation.cc:769] Sending connection context
0925 14:49:01.210478 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 14:49:01.210595 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260925 14:49:01.211256 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46863
I20260925 14:49:01.211325 903 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:01.211516 903 heartbeater.cc:511] Master 127.0.98.62:46863 requested a full tablet report, sending...
I20260925 14:49:01.211911 444 ts_manager.cc:194] Registered new tserver with Master: 721ea7baa5104b749100a45ac19c38c2 (127.0.98.3:42953)
I20260925 14:49:01.212312 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52651
I20260925 14:49:01.214423 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:01.236689 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:01.229096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.98.1:34491)
0925 14:49:01.229262 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:01.229267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:01.229277 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:01.229312 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:01.229314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:01.229328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:01.229395 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:01.229884 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.230647 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.231407 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.231701 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.231754 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:01.233872 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:01.233879 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:01.233879 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:01.233885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:01.235443 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.235841 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.235845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.235845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.235915 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.236123 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.236124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.236125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.236259 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:01.236412 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:01.236495 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 14:49:01.236570 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
W20260925 14:49:01.237116 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34320
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:01.258520 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:01.252053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34332 (local address 127.0.98.1:34491)
0925 14:49:01.252124 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:01.252125 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:01.252185 (+ 60us) server_negotiation.cc:407] Connection header received
0925 14:49:01.252228 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:01.252229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:01.252240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:01.252295 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:01.252818 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.253405 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.254077 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:01.254265 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:01.254316 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:01.256385 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:01.256386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:01.256387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:01.256394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:01.257557 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.257882 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.257883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.257884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.257940 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:01.258103 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:01.258103 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:01.258103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:01.258187 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:01.258273 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:01.258344 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:49:01.258456 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:49:01.259450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 506
I20260925 14:49:01.264083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 640
I20260925 14:49:01.269343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 774
I20260925 14:49:01.273452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 412
2026-09-25T14:49:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2234 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[917](info): set up 2 sockets
Sep 25 14:49:01 dist-test-slave-z2s5 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260925 14:49:02.309791 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:02.425571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38839
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38839
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:02.503199 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:02.503408 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:02.503446 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:02.504601 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:02.504642 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:02.504657 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:02.504668 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:02.504681 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:02.506155 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:02.506431 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:02.506657 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:02.511636 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:02.511631 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:02.511611 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:02.511948 930 server_base.cc:1034] running on GCE node
I20260925 14:49:02.512214 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:02.512470 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:02.513617 930 hybrid_clock.cc:648] HybridClock initialized: now 1790347742513606 us; error 28 us; skew 500 ppm
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:02.520356 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:02.520756 930 webserver.cc:460] Webserver started at http://127.0.98.62:44985/ using document root <none> and password file <none>
I20260925 14:49:02.520937 930 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:02.520977 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:02.521062 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:02.522081 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "931f81299cdf4bec93b213e550ace091"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "8f23f3080259d3a9c560509a7db6fc00"
server_key_iv: "c42f923c9514fa93fab6ebf4adcbd322"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.522372 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "931f81299cdf4bec93b213e550ace091"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "8f23f3080259d3a9c560509a7db6fc00"
server_key_iv: "c42f923c9514fa93fab6ebf4adcbd322"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.523710 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:02.524438 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.524607 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:02.524691 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "931f81299cdf4bec93b213e550ace091"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "8f23f3080259d3a9c560509a7db6fc00"
server_key_iv: "c42f923c9514fa93fab6ebf4adcbd322"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.524744 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:02.533653 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:02.535612 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:02.535720 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:02.539418 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38839
I20260925 14:49:02.539495 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38839 every 8 connection(s)
I20260925 14:49:02.539819 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:02.540678 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:02.542933 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091: Bootstrap starting.
I20260925 14:49:02.543516 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:02.543805 998 log.cc:824] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:02.544446 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091: No bootstrap required, opened a new log
I20260925 14:49:02.546008 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931f81299cdf4bec93b213e550ace091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38839 } }
I20260925 14:49:02.546140 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:02.546164 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931f81299cdf4bec93b213e550ace091, State: Initialized, Role: FOLLOWER
I20260925 14:49:02.546331 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931f81299cdf4bec93b213e550ace091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38839 } }
I20260925 14:49:02.546435 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:02.546479 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:02.546526 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:02.547075 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931f81299cdf4bec93b213e550ace091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38839 } }
I20260925 14:49:02.547183 998 leader_election.cc:302] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 931f81299cdf4bec93b213e550ace091; no voters:
I20260925 14:49:02.547358 998 leader_election.cc:288] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:02.547405 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:02.547612 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:02.547777 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [term 1 LEADER]: Becoming Leader. State: Replica: 931f81299cdf4bec93b213e550ace091, State: Running, Role: LEADER
I20260925 14:49:02.547875 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931f81299cdf4bec93b213e550ace091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38839 } }
I20260925 14:49:02.548252 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 931f81299cdf4bec93b213e550ace091. Latest consensus state: current_term: 1 leader_uuid: "931f81299cdf4bec93b213e550ace091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931f81299cdf4bec93b213e550ace091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38839 } } }
I20260925 14:49:02.548261 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "931f81299cdf4bec93b213e550ace091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "931f81299cdf4bec93b213e550ace091" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38839 } } }
I20260925 14:49:02.548357 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:02.548362 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:02.548555 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:02.549085 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:02.549917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 930
I20260925 14:49:02.549998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:02.550140 392 external_mini_cluster.cc:1428] Setting key a509d9222873f983ef4a7ab0579cd62a
I20260925 14:49:02.550767 1010 catalog_manager.cc:1347] Generated new cluster ID: f25f9254f7fe4526a67f9fc5006febf7
I20260925 14:49:02.550827 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:02.563458 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:02.550628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40822 (local address 127.0.98.62:38839)
0925 14:49:02.550769 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:02.550778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:02.554646 (+ 3868us) server_negotiation.cc:407] Connection header received
0925 14:49:02.554779 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:02.554785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:02.554837 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:02.554983 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:02.555602 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.556543 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.557304 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.557744 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.557886 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:02.560177 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:02.560181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:02.560181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:02.560200 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:02.561835 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:02.562226 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:02.562229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:02.562229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:02.562315 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:02.562552 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:02.562553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:02.562553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:02.562785 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:02.562954 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:02.563059 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:49:02.563202 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260925 14:49:02.563807 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:02.564369 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:02.571759 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 931f81299cdf4bec93b213e550ace091: Generated new TSK 0
I20260925 14:49:02.571944 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:02.600910 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38839
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:02.682497 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:02.682695 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:02.682718 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:02.683904 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:02.683944 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:02.684001 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:02.685475 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:02.685721 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:02.685942 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:02.690863 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:02.690912 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:02.690928 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:02.691175 1024 server_base.cc:1034] running on GCE node
I20260925 14:49:02.691393 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:02.691641 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:02.692808 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790347742692804 us; error 38 us; skew 500 ppm
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:02.695312 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:02.695701 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36661/ using document root <none> and password file <none>
I20260925 14:49:02.695922 1024 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:02.695968 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:02.696116 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:02.697146 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "d76bfd73b2874a6ba4e0f0c5516431cd"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "8cc360b3e5904323158b7eb2161349cc"
server_key_iv: "ebb528e3949c5923209cf97a4956d508"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.697510 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d76bfd73b2874a6ba4e0f0c5516431cd"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "8cc360b3e5904323158b7eb2161349cc"
server_key_iv: "ebb528e3949c5923209cf97a4956d508"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.698877 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:49:02.699584 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.699748 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:02.699843 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "d76bfd73b2874a6ba4e0f0c5516431cd"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "8cc360b3e5904323158b7eb2161349cc"
server_key_iv: "ebb528e3949c5923209cf97a4956d508"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.699921 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:02.713007 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:02.715364 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:02.715534 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:02.715781 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:02.716073 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:02.716127 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.716161 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:02.716188 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.722028 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37389
I20260925 14:49:02.722076 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37389 every 8 connection(s)
I20260925 14:49:02.722374 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:02.725714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1024
I20260925 14:49:02.725786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:02.725957 392 external_mini_cluster.cc:1428] Setting key a6e94a99cfba69093fa154983c3963e6
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:02.732535 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:02.723102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49569 (local address 127.0.98.62:38839)
0925 14:49:02.723172 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:02.723173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:02.723349 (+ 176us) server_negotiation.cc:407] Connection header received
0925 14:49:02.723810 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:02.723811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:02.723820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:02.723890 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:02.724744 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.725228 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.726373 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.726635 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.726713 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:02.729373 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:02.729375 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:02.729375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:02.729381 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:02.731152 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:02.731553 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:02.731554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:02.731555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:02.731619 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:02.731854 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:02.731855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:02.731855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:02.732013 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:02.732100 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:02.732338 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 14:49:02.732430 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:49:02.732574 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:02.723071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38839 (local address 127.0.98.1:49569)
0925 14:49:02.723215 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:02.723254 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:02.723729 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:02.723943 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:02.723946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:02.724484 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:02.724669 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:02.724673 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.725305 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.725306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:02.726290 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:02.726292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.726740 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.726741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:02.726900 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:02.727455 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:02.727465 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:02.729284 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:02.731250 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:02.731256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:02.731258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:02.731479 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:02.731691 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:02.731691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:02.731691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:02.731783 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:02.732092 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:02.732096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:02.732207 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 14:49:02.732289 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:49:02.732416 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260925 14:49:02.733366 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38839
I20260925 14:49:02.733465 1153 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:02.733659 1153 heartbeater.cc:511] Master 127.0.98.62:38839 requested a full tablet report, sending...
I20260925 14:49:02.734150 962 ts_manager.cc:194] Registered new tserver with Master: d76bfd73b2874a6ba4e0f0c5516431cd (127.0.98.1:37389)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:49:02.734870 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:02.750991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38839
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:02.833242 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:02.833482 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:02.833523 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:02.834715 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:02.834787 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:02.834832 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:02.836272 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:02.836544 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:02.836786 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:02.841681 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:02.841701 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:02.841840 1158 server_base.cc:1034] running on GCE node
W20260925 14:49:02.841701 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:02.842180 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:02.842484 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:02.843626 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790347742843610 us; error 39 us; skew 500 ppm
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:02.846601 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:02.847103 1158 webserver.cc:460] Webserver started at http://127.0.98.2:42935/ using document root <none> and password file <none>
I20260925 14:49:02.847373 1158 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:02.847445 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:02.847584 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:02.848781 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "00465a8b8eeb47358f09295c02f0652d"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "4bc8b5533748c3bd8b505c36380704f2"
server_key_iv: "a32d9f21d61f60ddb066a4428720617c"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.849150 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "00465a8b8eeb47358f09295c02f0652d"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "4bc8b5533748c3bd8b505c36380704f2"
server_key_iv: "a32d9f21d61f60ddb066a4428720617c"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.850584 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:02.851342 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.851524 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:02.851619 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "00465a8b8eeb47358f09295c02f0652d"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "4bc8b5533748c3bd8b505c36380704f2"
server_key_iv: "a32d9f21d61f60ddb066a4428720617c"
server_key_version: "encryptionkey@0"
I20260925 14:49:02.851709 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:02.868122 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:02.870715 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:02.870890 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:02.871132 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:02.871440 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:02.871496 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.871538 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:02.871567 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:02.877202 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42767
I20260925 14:49:02.877241 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42767 every 8 connection(s)
I20260925 14:49:02.877560 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:02.885682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1158
I20260925 14:49:02.885779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:02.885946 392 external_mini_cluster.cc:1428] Setting key 61e29f791d62e997a17a761c122d2ed8
I20260925 14:49:02.887132 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:02.878330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34329 (local address 127.0.98.62:38839)
0925 14:49:02.878380 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:02.878397 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:02.878567 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:49:02.879094 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:02.879096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:02.879108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:02.879176 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:02.880169 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.880675 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.881653 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.881848 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.881911 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:02.884056 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:02.884058 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:02.884058 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:02.884065 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:02.885520 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:02.886025 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:02.886026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:02.886027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:02.886112 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:02.886435 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:02.886436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:02.886436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:02.886563 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:02.886632 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:02.886990 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 14:49:02.887069 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:02.887442 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:02.878263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38839 (local address 127.0.98.2:34329)
0925 14:49:02.878446 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:02.878488 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:02.878961 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:02.879253 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:02.879257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:02.879837 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:02.880049 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:02.880055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.880763 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.880763 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:02.881563 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:02.881565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:02.881938 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:02.881938 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:02.882036 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:02.882609 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:02.882618 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:02.883970 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:02.885601 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:02.885607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:02.885610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:02.885937 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:02.886189 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:02.886189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:02.886190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:02.886345 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:02.886635 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:02.886641 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:02.886823 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 14:49:02.886964 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 14:49:02.887154 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260925 14:49:02.888545 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38839
I20260925 14:49:02.888641 1287 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:02.888824 1287 heartbeater.cc:511] Master 127.0.98.62:38839 requested a full tablet report, sending...
I20260925 14:49:02.889273 962 ts_manager.cc:194] Registered new tserver with Master: 00465a8b8eeb47358f09295c02f0652d (127.0.98.2:42767)
I20260925 14:49:02.889719 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34329
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:02.912830 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38839
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:02.991938 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:02.992170 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:02.992206 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:02.993326 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:02.993399 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:02.993448 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:02.994877 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38839
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:02.995127 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:02.995340 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:03.000295 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:03.000314 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:03.000295 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:03.000504 1292 server_base.cc:1034] running on GCE node
I20260925 14:49:03.000778 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:03.001019 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:03.002163 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790347743002155 us; error 32 us; skew 500 ppm
Sep 25 14:49:02 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:03.004757 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:03.005218 1292 webserver.cc:460] Webserver started at http://127.0.98.3:44959/ using document root <none> and password file <none>
I20260925 14:49:03.005439 1292 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:03.005484 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:03.005627 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:03.006722 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "9152018a971742708a8a0802c09a8c47"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "cc1803439ccaf0fc37c3c870305b1fb4"
server_key_iv: "847cf6f5443248bcc88f212dc367cfb6"
server_key_version: "encryptionkey@0"
I20260925 14:49:03.007071 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9152018a971742708a8a0802c09a8c47"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "cc1803439ccaf0fc37c3c870305b1fb4"
server_key_iv: "847cf6f5443248bcc88f212dc367cfb6"
server_key_version: "encryptionkey@0"
I20260925 14:49:03.008364 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:03.009092 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:03.009296 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:03.009366 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "9152018a971742708a8a0802c09a8c47"
format_stamp: "Formatted at 2026-09-25 14:49:02 on dist-test-slave-z2s5"
server_key: "cc1803439ccaf0fc37c3c870305b1fb4"
server_key_iv: "847cf6f5443248bcc88f212dc367cfb6"
server_key_version: "encryptionkey@0"
I20260925 14:49:03.009461 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:03.025246 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:03.027545 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:03.027706 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:03.027964 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:03.028301 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:03.028334 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:03.028384 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:03.028421 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:03.034200 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40801
I20260925 14:49:03.034242 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40801 every 8 connection(s)
I20260925 14:49:03.034626 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:03.037719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1292
I20260925 14:49:03.037835 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:03.037994 392 external_mini_cluster.cc:1428] Setting key e6322969b6e0dad61de9e25a1a71359e
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:03.044739 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.035421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42507 (local address 127.0.98.62:38839)
0925 14:49:03.035501 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.035504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.035688 (+ 184us) server_negotiation.cc:407] Connection header received
0925 14:49:03.036239 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.036241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.036253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.036311 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:03.037379 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.038125 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.039469 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.039668 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.039740 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.041867 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:03.041869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:03.041869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:03.041877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:03.043328 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.043765 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.043766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.043767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.043832 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.044087 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.044088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.044088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.044206 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:03.044284 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.044562 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.044646 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:49:03.044940 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.035383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38839 (local address 127.0.98.3:42507)
0925 14:49:03.035557 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:03.035605 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:03.036123 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:03.036389 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:03.036393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:03.037064 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:03.037262 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.037266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.038208 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.038209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.039365 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.039367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.039758 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.039759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.039892 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.040376 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:03.040385 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:03.041748 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:03.043432 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:03.043439 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:03.043441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:03.043672 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:03.043902 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:03.043903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:03.043903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:03.044018 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:03.044273 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:03.044279 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:03.044428 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 14:49:03.044540 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:49:03.044703 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
I20260925 14:49:03.045449 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38839
I20260925 14:49:03.045531 1421 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:03.045667 1421 heartbeater.cc:511] Master 127.0.98.62:38839 requested a full tablet report, sending...
I20260925 14:49:03.046015 962 ts_manager.cc:194] Registered new tserver with Master: 9152018a971742708a8a0802c09a8c47 (127.0.98.3:40801)
I20260925 14:49:03.046448 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42507
I20260925 14:49:03.049777 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:49:03.057418 1021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.051412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40830 (local address 127.0.98.62:38839)
0925 14:49:03.051457 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.051459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.051559 (+ 100us) server_negotiation.cc:407] Connection header received
0925 14:49:03.051611 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.051611 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.051619 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.051671 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:03.052237 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.052712 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.053446 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.053636 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.053691 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.054848 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:03.054850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:03.054850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:03.054856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:03.056368 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.056668 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.056669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.056670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.056728 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.056933 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.056933 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.056933 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.057072 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:03.057149 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.057227 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.057291 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:03.059653 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:03.061122 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:03.067479 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.063302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.0.1:57274)
0925 14:49:03.063429 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:03.063482 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:03.063541 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:03.064037 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:03.064041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:03.064043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:03.064114 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.064116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.065134 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.065135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.066267 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.066269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.067142 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.067143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.067230 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.067243 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:49:03.067344 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:49:03.067406 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260925 14:49:03.067487 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.063456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57274 (local address 127.0.98.2:42767)
0925 14:49:03.063841 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.063844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.063854 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:03.063908 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.063910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.063928 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.064020 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:03.064214 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.065026 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.066351 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.067045 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.067141 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.067200 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.067362 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.067406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":102,"threads_started":1}
I20260925 14:49:03.067889 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.063539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37389 (local address 127.0.0.1:58212)
0925 14:49:03.063782 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:03.063791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:03.063837 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:03.064364 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:03.064365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:03.064367 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:03.064400 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.064403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.065672 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.065673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.066720 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.066723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.067610 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.067610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.067688 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.067691 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:03.067742 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:49:03.067833 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":22,"thread_start_us":48,"threads_started":1}
I20260925 14:49:03.068112 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.063659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.0.98.1:37389)
0925 14:49:03.064135 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.064140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.064149 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:03.064185 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.064187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.064203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.064290 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:03.064487 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.065569 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.066793 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.067457 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.067523 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.067573 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.067997 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.068044 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":77,"threads_started":1}
I20260925 14:49:03.068429 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 52cf64ca9a394a16bf51c77d41d7e598 (DEFAULT_TABLE table=test-table [id=42aec1ef48094d9db12883fad669e2dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:03.068871 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52cf64ca9a394a16bf51c77d41d7e598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:03.068872 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 52cf64ca9a394a16bf51c77d41d7e598 (DEFAULT_TABLE table=test-table [id=42aec1ef48094d9db12883fad669e2dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:03.069200 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52cf64ca9a394a16bf51c77d41d7e598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:03.070577 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.063783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.0.98.3:40801)
0925 14:49:03.063972 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.063975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.064000 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:49:03.064074 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.064076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.064108 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.064179 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:03.064391 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.065525 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.066732 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.067451 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.067523 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.067573 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.070480 (+ 2907us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.070518 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260925 14:49:03.067848 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.063664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40801 (local address 127.0.0.1:55894)
0925 14:49:03.063943 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:03.063959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:03.064006 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:03.064263 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:03.064263 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:03.064266 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:03.064313 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.064315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.065635 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.065636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.066643 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.066645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.067609 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.067610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.067688 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.067693 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:03.067742 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:49:03.067782 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":4,"thread_start_us":69,"threads_started":1}
I20260925 14:49:03.071379 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 52cf64ca9a394a16bf51c77d41d7e598 (DEFAULT_TABLE table=test-table [id=42aec1ef48094d9db12883fad669e2dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:03.071678 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52cf64ca9a394a16bf51c77d41d7e598. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:03.072784 1436 tablet_bootstrap.cc:492] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: Bootstrap starting.
I20260925 14:49:03.073453 1436 tablet_bootstrap.cc:654] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:03.073998 1436 log.cc:824] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:03.074335 1435 tablet_bootstrap.cc:492] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: Bootstrap starting.
I20260925 14:49:03.074782 1436 tablet_bootstrap.cc:492] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: No bootstrap required, opened a new log
I20260925 14:49:03.074858 1436 ts_tablet_manager.cc:1397] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:03.074985 1438 tablet_bootstrap.cc:492] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: Bootstrap starting.
I20260925 14:49:03.075100 1435 tablet_bootstrap.cc:654] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:03.075570 1438 tablet_bootstrap.cc:654] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:03.075569 1435 log.cc:824] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:03.076176 1438 log.cc:824] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:03.076206 1435 tablet_bootstrap.cc:492] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: No bootstrap required, opened a new log
I20260925 14:49:03.076287 1435 ts_tablet_manager.cc:1397] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:49:03.076820 1436 raft_consensus.cc:348] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.076958 1436 raft_consensus.cc:374] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:03.076980 1436 raft_consensus.cc:729] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76bfd73b2874a6ba4e0f0c5516431cd, State: Initialized, Role: FOLLOWER
I20260925 14:49:03.077108 1436 consensus_queue.cc:260] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.077397 1436 ts_tablet_manager.cc:1428] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:49:03.077426 1153 heartbeater.cc:503] Master 127.0.98.62:38839 was elected leader, sending a full tablet report...
I20260925 14:49:03.078246 1435 raft_consensus.cc:348] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.078420 1435 raft_consensus.cc:374] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:03.078469 1435 raft_consensus.cc:729] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00465a8b8eeb47358f09295c02f0652d, State: Initialized, Role: FOLLOWER
I20260925 14:49:03.078585 1435 consensus_queue.cc:260] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.078874 1435 ts_tablet_manager.cc:1428] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:49:03.078877 1287 heartbeater.cc:503] Master 127.0.98.62:38839 was elected leader, sending a full tablet report...
I20260925 14:49:03.079551 1438 tablet_bootstrap.cc:492] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: No bootstrap required, opened a new log
I20260925 14:49:03.079627 1438 ts_tablet_manager.cc:1397] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:49:03.080976 1438 raft_consensus.cc:348] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.081127 1438 raft_consensus.cc:374] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:03.081172 1438 raft_consensus.cc:729] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9152018a971742708a8a0802c09a8c47, State: Initialized, Role: FOLLOWER
I20260925 14:49:03.081278 1438 consensus_queue.cc:260] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.081549 1421 heartbeater.cc:503] Master 127.0.98.62:38839 was elected leader, sending a full tablet report...
I20260925 14:49:03.081549 1438 ts_tablet_manager.cc:1428] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 14:49:03.128108 1288 tablet.cc:2369] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:49:03.223124 1154 tablet.cc:2369] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:49:03.285267 1422 tablet.cc:2369] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:03.389446 1443 raft_consensus.cc:482] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:03.389568 1443 raft_consensus.cc:504] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.389863 1443 leader_election.cc:288] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00465a8b8eeb47358f09295c02f0652d (127.0.98.2:42767), d76bfd73b2874a6ba4e0f0c5516431cd (127.0.98.1:37389)
I20260925 14:49:03.392861 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.390095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42767 (local address 127.0.98.3:50653)
0925 14:49:03.390159 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:03.390173 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:03.390268 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:03.390497 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:03.390498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:03.390506 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:03.390577 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.390579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.391262 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.391263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.392047 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.392049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.392597 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.392598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.392679 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.392682 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:03.392748 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:49:03.392788 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:49:03.392860 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.390206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50653 (local address 127.0.98.2:42767)
0925 14:49:03.390263 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.390265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.390278 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:49:03.390346 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.390347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.390361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.390463 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:03.390658 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.391161 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.392145 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.392506 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.392586 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.392597 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.392754 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.392789 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:49:03.392997 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.390336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56459 (local address 127.0.98.1:37389)
0925 14:49:03.390381 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.390384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.390537 (+ 153us) server_negotiation.cc:407] Connection header received
0925 14:49:03.390603 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.390604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.390617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.390670 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:03.390827 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.391274 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.392219 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.392661 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.392740 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.392749 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.392896 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.392932 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:03.393016 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.390258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37389 (local address 127.0.98.3:56459)
0925 14:49:03.390453 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:03.390478 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:03.390549 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:03.390682 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:03.390683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:03.390685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:03.390760 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.390761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.391361 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.391362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.392129 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:03.392131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.392739 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.392740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:03.392812 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.392815 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:03.392888 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:49:03.392932 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260925 14:49:03.393220 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cf64ca9a394a16bf51c77d41d7e598" candidate_uuid: "9152018a971742708a8a0802c09a8c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00465a8b8eeb47358f09295c02f0652d" is_pre_election: true
I20260925 14:49:03.393337 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cf64ca9a394a16bf51c77d41d7e598" candidate_uuid: "9152018a971742708a8a0802c09a8c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" is_pre_election: true
I20260925 14:49:03.393394 1241 raft_consensus.cc:2454] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9152018a971742708a8a0802c09a8c47 in term 0.
I20260925 14:49:03.393514 1107 raft_consensus.cc:2454] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9152018a971742708a8a0802c09a8c47 in term 0.
I20260925 14:49:03.393618 1308 leader_election.cc:302] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00465a8b8eeb47358f09295c02f0652d, 9152018a971742708a8a0802c09a8c47; no voters:
I20260925 14:49:03.393743 1443 raft_consensus.cc:2781] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:03.393811 1443 raft_consensus.cc:482] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:03.393829 1443 raft_consensus.cc:3037] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:03.394630 1443 raft_consensus.cc:504] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.394761 1443 leader_election.cc:288] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [CANDIDATE]: Term 1 election: Requested vote from peers 00465a8b8eeb47358f09295c02f0652d (127.0.98.2:42767), d76bfd73b2874a6ba4e0f0c5516431cd (127.0.98.1:37389)
I20260925 14:49:03.394930 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cf64ca9a394a16bf51c77d41d7e598" candidate_uuid: "9152018a971742708a8a0802c09a8c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00465a8b8eeb47358f09295c02f0652d"
I20260925 14:49:03.394958 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52cf64ca9a394a16bf51c77d41d7e598" candidate_uuid: "9152018a971742708a8a0802c09a8c47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd"
I20260925 14:49:03.395025 1241 raft_consensus.cc:3037] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:03.395023 1107 raft_consensus.cc:3037] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:03.395756 1107 raft_consensus.cc:2454] T 52cf64ca9a394a16bf51c77d41d7e598 P d76bfd73b2874a6ba4e0f0c5516431cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9152018a971742708a8a0802c09a8c47 in term 1.
I20260925 14:49:03.395768 1241 raft_consensus.cc:2454] T 52cf64ca9a394a16bf51c77d41d7e598 P 00465a8b8eeb47358f09295c02f0652d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9152018a971742708a8a0802c09a8c47 in term 1.
I20260925 14:49:03.395910 1308 leader_election.cc:302] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9152018a971742708a8a0802c09a8c47, d76bfd73b2874a6ba4e0f0c5516431cd; no voters:
I20260925 14:49:03.395995 1443 raft_consensus.cc:2781] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:03.396145 1443 raft_consensus.cc:686] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [term 1 LEADER]: Becoming Leader. State: Replica: 9152018a971742708a8a0802c09a8c47, State: Running, Role: LEADER
I20260925 14:49:03.396245 1443 consensus_queue.cc:237] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } }
I20260925 14:49:03.397076 962 catalog_manager.cc:5495] T 52cf64ca9a394a16bf51c77d41d7e598 P 9152018a971742708a8a0802c09a8c47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9152018a971742708a8a0802c09a8c47 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9152018a971742708a8a0802c09a8c47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00465a8b8eeb47358f09295c02f0652d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d76bfd73b2874a6ba4e0f0c5516431cd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9152018a971742708a8a0802c09a8c47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40801 } health_report { overall_health: HEALTHY } } }
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:03.423235 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.415795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.0.98.1:37389)
0925 14:49:03.415871 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.415873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.415928 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:49:03.415976 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.415977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.415986 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.416037 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:03.416581 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.417040 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.417723 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.417894 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.417942 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.420444 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:03.420462 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:03.420462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:03.420473 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:03.421961 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.422358 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.422362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.422363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.422465 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.422664 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.422665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.422665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.422829 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:03.422997 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.423037 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.423117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347743, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:03.446058 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.438902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.0.98.1:37389)
0925 14:49:03.438966 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.438969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.439096 (+ 127us) server_negotiation.cc:407] Connection header received
0925 14:49:03.439183 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.439184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.439200 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.439267 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:03.439853 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.440327 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.441084 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.441275 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.441362 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.443650 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:03.443652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:03.443652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:03.443660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:03.444826 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.445214 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.445216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.445216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.445271 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.445471 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.445472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.445472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.445573 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:03.445672 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.445866 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.445951 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:03.469470 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:03.462028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.98.1:37389)
0925 14:49:03.462124 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:03.462127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:03.462220 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:49:03.462296 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:03.462297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:03.462314 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:03.462379 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:03.462979 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.463464 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.464200 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:03.464389 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:03.464482 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:03.466864 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:03.466865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:03.466865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:03.466873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:03.468456 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.468762 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.468762 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.468762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.468855 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:03.469026 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:03.469026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:03.469026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:03.469121 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:03.469197 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:03.469291 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:49:03.469351 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 14:49:03.469975 1067 tablet_service.cc:2908] Rejecting scan request for tablet 52cf64ca9a394a16bf51c77d41d7e598: Uninitialized: clean time has not yet been initialized
I20260925 14:49:03.470844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1024
I20260925 14:49:03.476390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1158
I20260925 14:49:03.480953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1292
I20260925 14:49:03.485181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 930
2026-09-25T14:49:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2212 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[1459](info): set up 2 sockets
Sep 25 14:49:03 dist-test-slave-z2s5 krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260925 14:49:04.533924 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:04 dist-test-slave-z2s5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:04.647449 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41511
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:04.729159 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:04.729377 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:04.729401 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:04.730641 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:04.730687 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:49:04.730700 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:04.730713 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:04.730726 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:04.730738 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:04.732403 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:04.732684 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:04.732946 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:04.738230 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:04.738238 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:04.738238 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:04.738595 1472 server_base.cc:1034] running on GCE node
I20260925 14:49:04.738808 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:04.739061 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:04.740213 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790347744740199 us; error 31 us; skew 500 ppm
Sep 25 14:49:04 dist-test-slave-z2s5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:04.742880 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:04.743317 1472 webserver.cc:460] Webserver started at http://127.0.98.62:40151/ using document root <none> and password file <none>
I20260925 14:49:04.743539 1472 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:04.743584 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:04.743675 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:04.744717 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "186a79c77bef4291856867d22d1ff4ab"
format_stamp: "Formatted at 2026-09-25 14:49:04 on dist-test-slave-z2s5"
server_key: "a98044520a5acd31326c41dc9fc24aaa"
server_key_iv: "5f716fce8d86a9a4359813d0a3632859"
server_key_version: "encryptionkey@0"
I20260925 14:49:04.745020 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "186a79c77bef4291856867d22d1ff4ab"
format_stamp: "Formatted at 2026-09-25 14:49:04 on dist-test-slave-z2s5"
server_key: "a98044520a5acd31326c41dc9fc24aaa"
server_key_iv: "5f716fce8d86a9a4359813d0a3632859"
server_key_version: "encryptionkey@0"
I20260925 14:49:04.746304 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:49:04.747113 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:04.747277 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:04.747367 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "186a79c77bef4291856867d22d1ff4ab"
format_stamp: "Formatted at 2026-09-25 14:49:04 on dist-test-slave-z2s5"
server_key: "a98044520a5acd31326c41dc9fc24aaa"
server_key_iv: "5f716fce8d86a9a4359813d0a3632859"
server_key_version: "encryptionkey@0"
I20260925 14:49:04.747423 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:04.771294 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:04.773878 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:04.774000 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:04.778628 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41511
I20260925 14:49:04.778721 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41511 every 8 connection(s)
I20260925 14:49:04.779067 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:04.779830 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:04.782099 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: Bootstrap starting.
I20260925 14:49:04.782186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1472
I20260925 14:49:04.782305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:04.782516 392 external_mini_cluster.cc:1428] Setting key 83aa6e782070e71b18466bf6b5e86080
I20260925 14:49:04.782927 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:04.783561 1550 log.cc:824] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:04.784325 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: No bootstrap required, opened a new log
I20260925 14:49:04.786188 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a79c77bef4291856867d22d1ff4ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41511 } }
I20260925 14:49:04.786383 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:04.786449 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 186a79c77bef4291856867d22d1ff4ab, State: Initialized, Role: FOLLOWER
I20260925 14:49:04.786567 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a79c77bef4291856867d22d1ff4ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41511 } }
I20260925 14:49:04.786635 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:04.786660 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:04.786707 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:04.787292 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a79c77bef4291856867d22d1ff4ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41511 } }
I20260925 14:49:04.787389 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 186a79c77bef4291856867d22d1ff4ab; no voters:
I20260925 14:49:04.787577 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:04.787604 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:04.787715 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [term 1 LEADER]: Becoming Leader. State: Replica: 186a79c77bef4291856867d22d1ff4ab, State: Running, Role: LEADER
I20260925 14:49:04.787808 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a79c77bef4291856867d22d1ff4ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41511 } }
I20260925 14:49:04.787842 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:04.788298 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "186a79c77bef4291856867d22d1ff4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a79c77bef4291856867d22d1ff4ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41511 } } }
I20260925 14:49:04.788394 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:04.788298 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 186a79c77bef4291856867d22d1ff4ab. Latest consensus state: current_term: 1 leader_uuid: "186a79c77bef4291856867d22d1ff4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "186a79c77bef4291856867d22d1ff4ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41511 } } }
I20260925 14:49:04.788911 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:04.789075 1560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:04.789590 1560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:49:04 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:04.791410 1560 catalog_manager.cc:1347] Generated new cluster ID: c0d532a949f34f09a3257c65124c0fba
I20260925 14:49:04.791462 1560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:04.795143 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:04.783124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.62:41511)
0925 14:49:04.783440 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:04.783450 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:04.783500 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:49:04.784043 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:04.784053 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:04.784119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:04.784276 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:04.784961 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.786735 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.787695 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.788305 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.788415 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:04.791538 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:04.791542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:04.791543 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:04.791554 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:04.793286 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:04.793669 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:04.793673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:04.793673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:04.793761 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:04.793977 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:04.793978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:04.793979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:04.794217 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:04.794828 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:04.794895 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:49:04.795001 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":205,"threads_started":1}
I20260925 14:49:04.803934 1560 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:04.804481 1560 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:04.820145 1560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: Generated new TSK 0
I20260925 14:49:04.820359 1560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:04.851543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41511
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:04.932474 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:04.932686 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:04.932711 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:04.933884 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:04.933926 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:04.933947 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:04.933995 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:04.935554 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:04.935796 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:04.935995 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:04.940966 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:04.940930 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:04.940939 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:04.941222 1576 server_base.cc:1034] running on GCE node
I20260925 14:49:04.941429 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:04.941627 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:04.942761 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790347744942749 us; error 27 us; skew 500 ppm
Sep 25 14:49:04 dist-test-slave-z2s5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:04.945223 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:04.945637 1576 webserver.cc:460] Webserver started at http://127.0.98.1:39325/ using document root <none> and password file <none>
I20260925 14:49:04.945876 1576 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:04.945958 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:04.946099 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:04.947201 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a860375bcec4542800dd071bd96e183"
format_stamp: "Formatted at 2026-09-25 14:49:04 on dist-test-slave-z2s5"
server_key: "833ab4c331b61a567f76b1c94916b4a0"
server_key_iv: "ed4649d3414368bc028253c28806990a"
server_key_version: "encryptionkey@0"
I20260925 14:49:04.947561 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a860375bcec4542800dd071bd96e183"
format_stamp: "Formatted at 2026-09-25 14:49:04 on dist-test-slave-z2s5"
server_key: "833ab4c331b61a567f76b1c94916b4a0"
server_key_iv: "ed4649d3414368bc028253c28806990a"
server_key_version: "encryptionkey@0"
I20260925 14:49:04.948848 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:04.949527 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:04.949688 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:04.949786 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "2a860375bcec4542800dd071bd96e183"
format_stamp: "Formatted at 2026-09-25 14:49:04 on dist-test-slave-z2s5"
server_key: "833ab4c331b61a567f76b1c94916b4a0"
server_key_iv: "ed4649d3414368bc028253c28806990a"
server_key_version: "encryptionkey@0"
I20260925 14:49:04.949883 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:04.967650 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:04.970180 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:04.970358 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:04.970920 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:04.970981 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:04.971030 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:04.971062 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:49:04 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:04.978178 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33639
I20260925 14:49:04.978325 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33639 every 8 connection(s)
I20260925 14:49:04.978658 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:04.981400 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:04.971540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45531 (local address 127.0.98.62:41511)
0925 14:49:04.971589 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:04.971591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:04.971882 (+ 291us) server_negotiation.cc:407] Connection header received
0925 14:49:04.972530 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:04.972532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:04.972548 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:04.972627 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:04.973536 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.974425 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.975583 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.975879 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.975928 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:04.978565 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:04.978567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:04.978567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:04.978574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:04.980278 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:04.980657 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:04.980658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:04.980658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:04.980711 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:04.980904 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:04.980904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:04.980904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:04.981012 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:04.981073 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:04.981268 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:49:04.981337 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:04.981576 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:04.971536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.98.1:45531)
0925 14:49:04.971695 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:04.971739 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:04.972379 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:04.972723 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:04.972727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:04.973110 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:04.973412 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:04.973418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.974517 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.974519 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:04.975437 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:04.975439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.976018 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.976018 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:04.976127 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:04.976722 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:04.976730 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:04.978455 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:04.980364 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:04.980368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:04.980370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:04.980592 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:04.980767 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:04.980768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:04.980768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:04.980851 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:04.981064 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:04.981066 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:04.981174 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:49:04.981249 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:49:04.981365 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260925 14:49:04.982084 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41511
I20260925 14:49:04.982173 1707 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:04.982362 1707 heartbeater.cc:511] Master 127.0.98.62:41511 requested a full tablet report, sending...
I20260925 14:49:04.982945 1503 ts_manager.cc:194] Registered new tserver with Master: 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639)
I20260925 14:49:04.983685 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45531
I20260925 14:49:04.986569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1576
I20260925 14:49:04.986668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:04.987028 392 external_mini_cluster.cc:1428] Setting key a9109ee91b9c307c555c9be3633c9e8a
I20260925 14:49:04.989102 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:04.983276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49783 (local address 127.0.98.62:41511)
0925 14:49:04.983325 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:04.983327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:04.983391 (+ 64us) server_negotiation.cc:407] Connection header received
0925 14:49:04.983471 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:04.983472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:04.983507 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:04.983563 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:04.983921 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.984618 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.985323 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.985528 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.985618 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:04.986501 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:04.986503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:04.986503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:04.986510 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:04.987855 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:04.988327 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:04.988327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:04.988328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:04.988400 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:04.988624 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:04.988625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:04.988625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:04.988718 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:04.988769 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:04.988944 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:49:04.989046 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:49:04.989194 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:04.983203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.98.1:49783)
0925 14:49:04.983332 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:04.983344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:04.983408 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:04.983566 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:04.983567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:04.983761 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:04.983850 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:04.983852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.984707 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.984709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:04.985216 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:04.985218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:04.985620 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:04.985621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:04.985685 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:04.986069 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:04.986073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:04.986371 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:04.987976 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:04.987978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:04.987978 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:04.988244 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:04.988477 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:04.988478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:04.988479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:04.988557 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:04.988782 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:04.988782 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:04.988831 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:49:04.988911 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:49:04.989026 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:05.012688 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41511
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:05.095669 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:05.095898 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:05.095930 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:05.097049 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:05.097087 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:05.097107 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:05.097124 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:05.098906 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:05.099119 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:05.099309 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:05.104537 1716 server_base.cc:1034] running on GCE node
W20260925 14:49:05.104558 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:05.104534 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:05.104534 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:05.105022 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:05.105250 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:05.106451 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790347745106430 us; error 40 us; skew 500 ppm
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:05.109004 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:05.109570 1716 webserver.cc:460] Webserver started at http://127.0.98.2:45933/ using document root <none> and password file <none>
I20260925 14:49:05.109781 1716 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:05.109823 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:05.109915 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:05.111073 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c1268bbffac4f138d8afe23c5c5cc0f"
format_stamp: "Formatted at 2026-09-25 14:49:05 on dist-test-slave-z2s5"
server_key: "ba94dfee909b8662c66e75fdb19f343c"
server_key_iv: "3cfc2beaa8ec2bf2d564c9eb17b928d1"
server_key_version: "encryptionkey@0"
I20260925 14:49:05.111372 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c1268bbffac4f138d8afe23c5c5cc0f"
format_stamp: "Formatted at 2026-09-25 14:49:05 on dist-test-slave-z2s5"
server_key: "ba94dfee909b8662c66e75fdb19f343c"
server_key_iv: "3cfc2beaa8ec2bf2d564c9eb17b928d1"
server_key_version: "encryptionkey@0"
I20260925 14:49:05.112634 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:05.113337 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.113554 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:05.113638 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "9c1268bbffac4f138d8afe23c5c5cc0f"
format_stamp: "Formatted at 2026-09-25 14:49:05 on dist-test-slave-z2s5"
server_key: "ba94dfee909b8662c66e75fdb19f343c"
server_key_iv: "3cfc2beaa8ec2bf2d564c9eb17b928d1"
server_key_version: "encryptionkey@0"
I20260925 14:49:05.113696 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:05.129927 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:05.131906 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:05.132035 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:05.132484 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:05.132521 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.132542 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:05.132560 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.139237 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39147
I20260925 14:49:05.139298 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39147 every 8 connection(s)
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:05.139645 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:05.142375 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.133030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55725 (local address 127.0.98.62:41511)
0925 14:49:05.133104 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.133106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.133262 (+ 156us) server_negotiation.cc:407] Connection header received
0925 14:49:05.133747 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.133748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.133761 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.133840 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:05.134712 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.135202 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.136818 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.137175 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.137286 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.139869 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:05.139870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:05.139871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:05.139880 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:05.141197 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.141601 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.141601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.141602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.141651 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.141865 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.141865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.141865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.141975 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:05.142029 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.142250 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.142312 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:49:05.142663 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.132987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.98.2:55725)
0925 14:49:05.133136 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.133175 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.133655 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.133935 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.133944 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.134322 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:05.134617 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.134622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.135295 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.135296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.136709 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.136712 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.137305 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.137306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.137470 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.138022 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:05.138032 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:05.139767 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:05.141288 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.141294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.141296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.141530 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.141710 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.141711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.141711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.141805 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.142030 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:05.142034 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:05.142144 (+ 110us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.142224 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.142342 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 14:49:05.143165 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41511
I20260925 14:49:05.143246 1847 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:05.143435 1847 heartbeater.cc:511] Master 127.0.98.62:41511 requested a full tablet report, sending...
I20260925 14:49:05.143850 1504 ts_manager.cc:194] Registered new tserver with Master: 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147)
I20260925 14:49:05.144359 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55725
I20260925 14:49:05.147697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1716
I20260925 14:49:05.147797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:05.148018 392 external_mini_cluster.cc:1428] Setting key 90bef5c4bab1ac48ec445fd79bb51e16
I20260925 14:49:05.150154 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.144228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33823 (local address 127.0.98.62:41511)
0925 14:49:05.144268 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.144270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.144330 (+ 60us) server_negotiation.cc:407] Connection header received
0925 14:49:05.144394 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.144395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.144403 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.144488 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:05.144872 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.145336 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.146100 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.146313 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.146438 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.147277 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:05.147278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:05.147279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:05.147285 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:05.148939 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.149268 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.149269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.149270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.149328 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.149507 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.149508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.149509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.149699 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:05.149785 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.149951 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.150028 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:05.150166 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.144147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.98.2:33823)
0925 14:49:05.144272 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.144282 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.144340 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.144513 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.144514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.144722 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:05.144792 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.144795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.145426 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.145427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.145949 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.145951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.146464 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.146465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.146531 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.146930 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:05.146934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:05.147201 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:05.149030 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.149031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.149031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.149199 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.149398 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.149399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.149400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.149451 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.149786 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:05.149787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:05.149831 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.149923 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.150001 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:05.172658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41511
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:05.255955 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:05.256166 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:05.256191 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:05.257366 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:05.257406 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:05.257426 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:05.257444 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:05.258930 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:05.259114 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:05.259291 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:05.264271 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:05.264262 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:05.264262 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:05.264508 1856 server_base.cc:1034] running on GCE node
I20260925 14:49:05.264703 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:05.264974 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:05.266172 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790347745266155 us; error 28 us; skew 500 ppm
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:05.268677 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:05.269172 1856 webserver.cc:460] Webserver started at http://127.0.98.3:39511/ using document root <none> and password file <none>
I20260925 14:49:05.269408 1856 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:05.269482 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:05.269641 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:05.270797 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "6481f64f2ec0440792a4c5dc068d68a7"
format_stamp: "Formatted at 2026-09-25 14:49:05 on dist-test-slave-z2s5"
server_key: "fca19982b09e81ff6c4ac1d73a6f6ee9"
server_key_iv: "d9788350dfbbed42399f696f9ba5baca"
server_key_version: "encryptionkey@0"
I20260925 14:49:05.271178 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6481f64f2ec0440792a4c5dc068d68a7"
format_stamp: "Formatted at 2026-09-25 14:49:05 on dist-test-slave-z2s5"
server_key: "fca19982b09e81ff6c4ac1d73a6f6ee9"
server_key_iv: "d9788350dfbbed42399f696f9ba5baca"
server_key_version: "encryptionkey@0"
I20260925 14:49:05.272509 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:05.273253 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.273432 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:05.273537 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "6481f64f2ec0440792a4c5dc068d68a7"
format_stamp: "Formatted at 2026-09-25 14:49:05 on dist-test-slave-z2s5"
server_key: "fca19982b09e81ff6c4ac1d73a6f6ee9"
server_key_iv: "d9788350dfbbed42399f696f9ba5baca"
server_key_version: "encryptionkey@0"
I20260925 14:49:05.273622 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:05.282723 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:05.284745 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:05.284937 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:05.285576 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:05.285627 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.285661 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:05.285682 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:05.293819 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42041
I20260925 14:49:05.293898 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42041 every 8 connection(s)
I20260925 14:49:05.294248 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:05.296692 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.286004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41761 (local address 127.0.98.62:41511)
0925 14:49:05.286079 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.286082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.286315 (+ 233us) server_negotiation.cc:407] Connection header received
0925 14:49:05.286890 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.286892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.286906 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.286998 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:05.287710 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.288382 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.290022 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.290263 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.290360 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.293994 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:05.293996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:05.293997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:05.294006 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:05.295537 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.295935 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.295936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.295936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.295991 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.296214 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.296215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.296215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.296323 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:05.296373 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.296569 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.296628 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:49:05.296873 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.286009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.98.3:41761)
0925 14:49:05.286182 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.286225 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.286783 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.287053 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.287056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.287294 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:05.287611 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.287616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.288507 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.288508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.289941 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.289944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.290370 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.290372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.290582 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.291184 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:05.291193 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:05.293884 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:05.295617 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.295622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.295623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.295851 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.296053 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.296053 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.296054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.296156 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.296364 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:05.296366 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:05.296482 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.296559 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.296680 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":72,"threads_started":1}
I20260925 14:49:05.297323 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41511
I20260925 14:49:05.297417 1987 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:05.297610 1987 heartbeater.cc:511] Master 127.0.98.62:41511 requested a full tablet report, sending...
I20260925 14:49:05.297664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 1856
I20260925 14:49:05.297737 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:05.297930 392 external_mini_cluster.cc:1428] Setting key d68bb3a89ab4abd54660ebfd104544c3
I20260925 14:49:05.298039 1503 ts_manager.cc:194] Registered new tserver with Master: 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3:42041)
I20260925 14:49:05.298593 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41761
I20260925 14:49:05.298915 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:49:05.305074 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.298464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44097 (local address 127.0.98.62:41511)
0925 14:49:05.298506 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.298508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.298647 (+ 139us) server_negotiation.cc:407] Connection header received
0925 14:49:05.298768 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.298769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.298779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.298845 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:05.299229 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.299733 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.300836 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.301088 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.301123 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.302173 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:05.302175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:05.302175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:05.302181 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:05.303895 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.304217 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.304219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.304219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.304281 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.304473 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.304474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.304475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.304619 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:05.304695 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.304889 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.304952 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:05.305138 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.298457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.98.3:44097)
0925 14:49:05.298588 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.298598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.298693 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.298855 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.298856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.299077 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:05.299139 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.299141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.299844 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.299845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.300734 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.300736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.301185 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.301186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.301232 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.301668 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:05.301673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:05.302087 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:05.303987 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.303988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.303988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.304144 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.304348 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.304349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.304349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.304398 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.304704 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:05.304705 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:05.304759 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.304846 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.304997 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:05.333004 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.324085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.98.62:41511)
0925 14:49:05.325249 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.325252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.325264 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:49:05.325313 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.325314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.325333 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.325452 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:05.326281 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.327242 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.328105 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.328286 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.328351 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.330722 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:05.330724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:05.330724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:05.330730 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:05.332045 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.332379 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.332379 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.332380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.332426 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.332588 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.332588 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.332588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.332676 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:05.332749 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.332879 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.332938 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1130}
I20260925 14:49:05.334659 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:05.336305 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:05.342864 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.338851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33639 (local address 127.0.0.1:46982)
0925 14:49:05.339155 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.339166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.339221 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.339517 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.339517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.339520 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:05.339570 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.339572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.340497 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.340498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.341716 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.341718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.342560 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.342640 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.342655 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.342728 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.342780 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":76,"threads_started":1}
I20260925 14:49:05.342883 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.338964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.0.98.1:33639)
0925 14:49:05.339152 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.339158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.339286 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:49:05.339346 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.339349 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.339365 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.339446 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:05.339648 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.340407 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.341809 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.342381 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342551 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.342640 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.342743 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.342780 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":85,"threads_started":1}
I20260925 14:49:05.343183 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.339243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.0.98.3:42041)
0925 14:49:05.339433 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.339437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.339471 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:49:05.339550 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.339553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.339569 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.339643 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:05.339820 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.340900 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.341919 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.342704 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342768 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.342841 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.343063 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.343097 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260925 14:49:05.343359 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.338950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52124 (local address 127.0.98.2:39147)
0925 14:49:05.339242 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.339245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.339253 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:05.339283 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.339285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.339297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.339398 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:05.339668 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.340805 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342005 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.342697 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342768 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.342842 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.343214 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.343259 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":84,"threads_started":1}
I20260925 14:49:05.343127 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.338840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39147 (local address 127.0.0.1:52124)
0925 14:49:05.339002 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.339046 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.339112 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.339481 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.339484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.339486 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:05.339581 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.339584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.340914 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.340915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.341942 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.341944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.342891 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.342953 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.342955 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.343002 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.343055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":2,"thread_start_us":54,"threads_started":1}
I20260925 14:49:05.343787 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 96026b1c94ca4203a056e9dce1c6575d (DEFAULT_TABLE table=test-table [id=98bc947e547242169143f3b77288a921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:05.344192 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96026b1c94ca4203a056e9dce1c6575d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.344107 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 96026b1c94ca4203a056e9dce1c6575d (DEFAULT_TABLE table=test-table [id=98bc947e547242169143f3b77288a921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:05.344424 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96026b1c94ca4203a056e9dce1c6575d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.347357 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.339053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42041 (local address 127.0.0.1:44964)
0925 14:49:05.339419 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.339428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.339479 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.339711 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.339712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.339714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:05.339747 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.339748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.340964 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.340966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.341830 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.341832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.342877 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.342878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.342945 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.342950 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.343002 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.343279 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":227,"thread_start_us":64,"threads_started":1}
I20260925 14:49:05.347365 2010 tablet_bootstrap.cc:492] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Bootstrap starting.
I20260925 14:49:05.348011 2010 tablet_bootstrap.cc:654] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.348059 2011 tablet_bootstrap.cc:492] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Bootstrap starting.
I20260925 14:49:05.348091 1920 tablet_service.cc:1467] Processing CreateTablet for tablet 96026b1c94ca4203a056e9dce1c6575d (DEFAULT_TABLE table=test-table [id=98bc947e547242169143f3b77288a921]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:05.348347 2010 log.cc:824] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:05.348423 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96026b1c94ca4203a056e9dce1c6575d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.348658 2011 tablet_bootstrap.cc:654] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.348994 2011 log.cc:824] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:05.349094 2010 tablet_bootstrap.cc:492] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: No bootstrap required, opened a new log
I20260925 14:49:05.349180 2010 ts_tablet_manager.cc:1397] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:05.349753 2011 tablet_bootstrap.cc:492] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: No bootstrap required, opened a new log
I20260925 14:49:05.349861 2011 ts_tablet_manager.cc:1397] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:05.351109 2010 raft_consensus.cc:348] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.351239 2010 raft_consensus.cc:374] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.351260 2010 raft_consensus.cc:729] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a860375bcec4542800dd071bd96e183, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.351416 2010 consensus_queue.cc:260] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.351681 2010 ts_tablet_manager.cc:1428] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:49:05.351809 1707 heartbeater.cc:503] Master 127.0.98.62:41511 was elected leader, sending a full tablet report...
I20260925 14:49:05.352176 2015 tablet_bootstrap.cc:492] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Bootstrap starting.
I20260925 14:49:05.352532 2011 raft_consensus.cc:348] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.352721 2011 raft_consensus.cc:374] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.352773 2011 raft_consensus.cc:729] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1268bbffac4f138d8afe23c5c5cc0f, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.352833 2015 tablet_bootstrap.cc:654] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.352921 2011 consensus_queue.cc:260] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.353130 2015 log.cc:824] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:05.353228 1847 heartbeater.cc:503] Master 127.0.98.62:41511 was elected leader, sending a full tablet report...
I20260925 14:49:05.353459 2011 ts_tablet_manager.cc:1428] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:49:05.353917 2015 tablet_bootstrap.cc:492] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: No bootstrap required, opened a new log
I20260925 14:49:05.353999 2015 ts_tablet_manager.cc:1397] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:05.355504 2015 raft_consensus.cc:348] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.355645 2015 raft_consensus.cc:374] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.355690 2015 raft_consensus.cc:729] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6481f64f2ec0440792a4c5dc068d68a7, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.355795 2015 consensus_queue.cc:260] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.356034 2015 ts_tablet_manager.cc:1428] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:49:05.356103 1987 heartbeater.cc:503] Master 127.0.98.62:41511 was elected leader, sending a full tablet report...
W20260925 14:49:05.390211 1848 tablet.cc:2369] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:49:05.479406 1708 tablet.cc:2369] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:49:05.544893 1988 tablet.cc:2369] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:05.707417 2018 raft_consensus.cc:482] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:05.707540 2018 raft_consensus.cc:504] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.708021 2018 leader_election.cc:288] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639), 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147)
I20260925 14:49:05.710855 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.708302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43837 (local address 127.0.98.2:39147)
0925 14:49:05.708352 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.708354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.708365 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:05.708395 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.708396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.708409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.708490 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:05.708644 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.709112 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.710039 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.710495 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.710581 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.710591 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.710738 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.710772 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:49:05.710887 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.708220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39147 (local address 127.0.98.3:43837)
0925 14:49:05.708284 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.708296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.708341 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.708492 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.708493 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.708495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:05.708571 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.708573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.709201 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.709202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.709947 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.709949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.710582 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.710583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.710666 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.710668 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.710734 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.710805 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:49:05.711004 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.708244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56523 (local address 127.0.98.1:33639)
0925 14:49:05.708288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.708291 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.708488 (+ 197us) server_negotiation.cc:407] Connection header received
0925 14:49:05.708564 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.708566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.708578 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.708649 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:05.708817 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.709269 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.710215 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.710662 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.710741 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.710751 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.710906 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.710943 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:49:05.711024 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.708221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33639 (local address 127.0.98.3:56523)
0925 14:49:05.708397 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.708408 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.708497 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.708651 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.708652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.708653 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:05.708733 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.708734 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.709366 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.709367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.710128 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.710130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.710753 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.710754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.710824 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.710827 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.710888 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.710926 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":11,"thread_start_us":106,"threads_started":1}
I20260925 14:49:05.711230 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96026b1c94ca4203a056e9dce1c6575d" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" is_pre_election: true
I20260925 14:49:05.711387 1801 raft_consensus.cc:2454] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 0.
I20260925 14:49:05.711374 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96026b1c94ca4203a056e9dce1c6575d" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a860375bcec4542800dd071bd96e183" is_pre_election: true
I20260925 14:49:05.711496 1661 raft_consensus.cc:2454] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 0.
I20260925 14:49:05.711658 1872 leader_election.cc:302] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6481f64f2ec0440792a4c5dc068d68a7, 9c1268bbffac4f138d8afe23c5c5cc0f; no voters:
I20260925 14:49:05.711782 2018 raft_consensus.cc:2781] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:05.711838 2018 raft_consensus.cc:482] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:05.711854 2018 raft_consensus.cc:3037] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:05.759495 2018 raft_consensus.cc:504] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.759733 2018 leader_election.cc:288] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 election: Requested vote from peers 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639), 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147)
I20260925 14:49:05.760000 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96026b1c94ca4203a056e9dce1c6575d" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a860375bcec4542800dd071bd96e183"
I20260925 14:49:05.760026 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96026b1c94ca4203a056e9dce1c6575d" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f"
I20260925 14:49:05.760123 1661 raft_consensus.cc:3037] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:05.760150 1801 raft_consensus.cc:3037] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:05.760847 1801 raft_consensus.cc:2454] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 1.
I20260925 14:49:05.761086 1872 leader_election.cc:302] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6481f64f2ec0440792a4c5dc068d68a7, 9c1268bbffac4f138d8afe23c5c5cc0f; no voters:
I20260925 14:49:05.761200 2018 raft_consensus.cc:2781] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:05.761206 1661 raft_consensus.cc:2454] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 1.
I20260925 14:49:05.761368 2018 raft_consensus.cc:686] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6481f64f2ec0440792a4c5dc068d68a7, State: Running, Role: LEADER
I20260925 14:49:05.761497 2018 consensus_queue.cc:237] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.762521 1504 catalog_manager.cc:5495] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6481f64f2ec0440792a4c5dc068d68a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } health_report { overall_health: HEALTHY } } }
I20260925 14:49:05.783581 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.780919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.98.3:42041)
0925 14:49:05.780992 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.780994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.781004 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:05.781148 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.781154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.781173 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.781250 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:05.781396 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.781863 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.782746 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.782937 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.783024 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.783222 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:05.783320 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:05.783418 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.783466 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.783487 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:49:05.785451 1661 raft_consensus.cc:1261] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Refusing update from remote peer 6481f64f2ec0440792a4c5dc068d68a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:05.785480 1801 raft_consensus.cc:1261] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Refusing update from remote peer 6481f64f2ec0440792a4c5dc068d68a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:05.785779 2018 consensus_queue.cc:1035] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:05.786348 2020 consensus_queue.cc:1035] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:05.788800 2023 mvcc.cc:205] Tried to move back new op lower bound from 7333264366734041088 to 7333264366639706112. Current Snapshot: MvccSnapshot[applied={T|T < 7333264366734041088}]
I20260925 14:49:05.789227 2024 mvcc.cc:205] Tried to move back new op lower bound from 7333264366734041088 to 7333264366639706112. Current Snapshot: MvccSnapshot[applied={T|T < 7333264366734041088}]
I20260925 14:49:05.789319 2025 mvcc.cc:205] Tried to move back new op lower bound from 7333264366734041088 to 7333264366639706112. Current Snapshot: MvccSnapshot[applied={T|T < 7333264366734041088}]
I20260925 14:49:05.792820 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:05.792920 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:05.793651 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: Sending DeleteTablet for 3 replicas of tablet 96026b1c94ca4203a056e9dce1c6575d
I20260925 14:49:05.793982 1920 tablet_service.cc:1514] Processing DeleteTablet for tablet 96026b1c94ca4203a056e9dce1c6575d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44964
I20260925 14:49:05.793989 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 96026b1c94ca4203a056e9dce1c6575d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46982
I20260925 14:49:05.794003 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 96026b1c94ca4203a056e9dce1c6575d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52124
I20260925 14:49:05.794198 2033 tablet_replica.cc:330] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: stopping tablet replica
I20260925 14:49:05.794304 2033 raft_consensus.cc:2229] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:05.794440 2033 raft_consensus.cc:2258] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:05.794557 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:05.794710 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:05.795078 2035 tablet_replica.cc:330] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: stopping tablet replica
I20260925 14:49:05.795184 2035 raft_consensus.cc:2229] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:05.795219 2033 ts_tablet_manager.cc:1905] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:05.795349 2035 raft_consensus.cc:2258] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:05.795523 2034 tablet_replica.cc:330] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: stopping tablet replica
I20260925 14:49:05.795635 2034 raft_consensus.cc:2229] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:05.795747 2034 raft_consensus.cc:2258] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:05.796260 2034 ts_tablet_manager.cc:1905] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:05.796552 1920 tablet_service.cc:1467] Processing CreateTablet for tablet 79289343173348abaa8f4c6026d5cf86 (DEFAULT_TABLE table=test-table [id=458817f3cded4728aaf145acf6ccdec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:05.796602 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 79289343173348abaa8f4c6026d5cf86 (DEFAULT_TABLE table=test-table [id=458817f3cded4728aaf145acf6ccdec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:05.796667 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79289343173348abaa8f4c6026d5cf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.797079 2035 ts_tablet_manager.cc:1905] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:05.798089 2015 tablet_bootstrap.cc:492] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Bootstrap starting.
I20260925 14:49:05.798079 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 79289343173348abaa8f4c6026d5cf86 (DEFAULT_TABLE table=test-table [id=458817f3cded4728aaf145acf6ccdec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:05.798535 2015 tablet_bootstrap.cc:654] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.799119 2015 tablet_bootstrap.cc:492] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: No bootstrap required, opened a new log
I20260925 14:49:05.799168 2015 ts_tablet_manager.cc:1397] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:05.799273 2015 raft_consensus.cc:348] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.799324 2015 raft_consensus.cc:374] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.799336 2015 raft_consensus.cc:729] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6481f64f2ec0440792a4c5dc068d68a7, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.799396 2015 consensus_queue.cc:260] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.799517 2015 ts_tablet_manager.cc:1428] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.844738 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79289343173348abaa8f4c6026d5cf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.845386 2035 ts_tablet_manager.cc:1918] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:05.845482 2035 log.cc:1196] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/96026b1c94ca4203a056e9dce1c6575d
I20260925 14:49:05.845749 2035 ts_tablet_manager.cc:1939] T 96026b1c94ca4203a056e9dce1c6575d P 6481f64f2ec0440792a4c5dc068d68a7: Deleting consensus metadata
I20260925 14:49:05.846235 2010 tablet_bootstrap.cc:492] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Bootstrap starting.
I20260925 14:49:05.846510 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79289343173348abaa8f4c6026d5cf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.846712 2010 tablet_bootstrap.cc:654] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.848071 2033 ts_tablet_manager.cc:1918] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:05.848145 2033 log.cc:1196] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/96026b1c94ca4203a056e9dce1c6575d
I20260925 14:49:05.848460 2033 ts_tablet_manager.cc:1939] T 96026b1c94ca4203a056e9dce1c6575d P 2a860375bcec4542800dd071bd96e183: Deleting consensus metadata
I20260925 14:49:05.848906 2011 tablet_bootstrap.cc:492] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Bootstrap starting.
I20260925 14:49:05.849231 2010 tablet_bootstrap.cc:492] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: No bootstrap required, opened a new log
I20260925 14:49:05.849291 2010 ts_tablet_manager.cc:1397] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:49:05.849327 2011 tablet_bootstrap.cc:654] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.849547 2034 ts_tablet_manager.cc:1918] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:05.849665 2030 raft_consensus.cc:482] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:05.849649 2010 raft_consensus.cc:348] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.849725 2010 raft_consensus.cc:374] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.849743 2010 raft_consensus.cc:729] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a860375bcec4542800dd071bd96e183, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.849735 2030 raft_consensus.cc:504] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.849793 2010 consensus_queue.cc:260] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.849879 2030 leader_election.cc:288] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639), 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147)
I20260925 14:49:05.849891 2010 ts_tablet_manager.cc:1428] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:05.849998 2034 log.cc:1196] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/96026b1c94ca4203a056e9dce1c6575d
I20260925 14:49:05.850153 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79289343173348abaa8f4c6026d5cf86" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" is_pre_election: true
I20260925 14:49:05.850183 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79289343173348abaa8f4c6026d5cf86" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a860375bcec4542800dd071bd96e183" is_pre_election: true
I20260925 14:49:05.850240 1661 raft_consensus.cc:2454] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 0.
W20260925 14:49:05.850486 1872 leader_election.cc:341] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147): Illegal state: must be running to vote when last-logged opid is not known
I20260925 14:49:05.850544 2011 tablet_bootstrap.cc:492] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: No bootstrap required, opened a new log
I20260925 14:49:05.850553 1872 leader_election.cc:302] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a860375bcec4542800dd071bd96e183, 6481f64f2ec0440792a4c5dc068d68a7; no voters: 9c1268bbffac4f138d8afe23c5c5cc0f
I20260925 14:49:05.850584 2011 ts_tablet_manager.cc:1397] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:05.850672 2030 raft_consensus.cc:2781] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:05.850698 2030 raft_consensus.cc:482] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:05.850718 2030 raft_consensus.cc:3037] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:05.850744 2011 raft_consensus.cc:348] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.850811 2011 raft_consensus.cc:374] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.850833 2011 raft_consensus.cc:729] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1268bbffac4f138d8afe23c5c5cc0f, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.850886 2011 consensus_queue.cc:260] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.850991 2011 ts_tablet_manager.cc:1428] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:05.851540 2030 raft_consensus.cc:504] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.851702 2030 leader_election.cc:288] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 election: Requested vote from peers 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639), 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147)
I20260925 14:49:05.851941 2034 ts_tablet_manager.cc:1939] T 96026b1c94ca4203a056e9dce1c6575d P 9c1268bbffac4f138d8afe23c5c5cc0f: Deleting consensus metadata
I20260925 14:49:05.851912 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79289343173348abaa8f4c6026d5cf86" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a860375bcec4542800dd071bd96e183"
I20260925 14:49:05.851982 1661 raft_consensus.cc:3037] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:05.852102 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79289343173348abaa8f4c6026d5cf86" candidate_uuid: "6481f64f2ec0440792a4c5dc068d68a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f"
I20260925 14:49:05.852169 1801 raft_consensus.cc:3037] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:05.852728 1661 raft_consensus.cc:2454] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 1.
I20260925 14:49:05.852767 1801 raft_consensus.cc:2454] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6481f64f2ec0440792a4c5dc068d68a7 in term 1.
I20260925 14:49:05.852931 1874 leader_election.cc:302] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a860375bcec4542800dd071bd96e183, 6481f64f2ec0440792a4c5dc068d68a7; no voters:
I20260925 14:49:05.853024 2030 raft_consensus.cc:2781] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:05.853088 2030 raft_consensus.cc:686] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6481f64f2ec0440792a4c5dc068d68a7, State: Running, Role: LEADER
I20260925 14:49:05.853139 2030 consensus_queue.cc:237] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.853629 1504 catalog_manager.cc:5495] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6481f64f2ec0440792a4c5dc068d68a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } health_report { overall_health: HEALTHY } } }
I20260925 14:49:05.856920 1491 catalog_manager.cc:4841] TS 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3:42041): tablet 96026b1c94ca4203a056e9dce1c6575d (table test-table [id=98bc947e547242169143f3b77288a921]) successfully deleted
Sep 25 14:49:05 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:05.891806 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.883195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41511 (local address 127.0.0.1:57850)
0925 14:49:05.883358 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:05.883371 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:05.883433 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:05.883659 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:05.883660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:05.884059 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:05.884106 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.884108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.885135 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.885137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.885736 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:05.885738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.886241 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.886242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:05.886300 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.886899 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:05.886903 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:05.888657 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:05.890632 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.890639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.890640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.890917 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.891155 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:05.891155 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:05.891155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:05.891211 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:05.891488 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:05.891493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:05.891535 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:49:05.891587 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:49:05.891677 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260925 14:49:05.891884 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:05.883218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.0.98.62:41511)
0925 14:49:05.883429 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:05.883431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:05.883446 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:49:05.883533 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:05.883534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:05.883547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:05.883623 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:05.884202 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.885040 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.885822 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:05.886152 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:05.886227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:05.888763 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:05.888765 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:05.888765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:05.888773 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:05.890559 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.891008 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.891009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.891010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.891083 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:05.891273 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:05.891274 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:05.891274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:05.891417 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:05.891491 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:05.891632 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:49:05.891746 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":172,"thread_start_us":79,"threads_started":1}
I20260925 14:49:05.894352 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57850:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:49:05.894613 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:05.896055 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 33da144aed6347ddbef4eca13883aabb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=576d9c0fc616435386b9001151311541]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:05.896103 1920 tablet_service.cc:1467] Processing CreateTablet for tablet 33da144aed6347ddbef4eca13883aabb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=576d9c0fc616435386b9001151311541]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:05.896217 1920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33da144aed6347ddbef4eca13883aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.896219 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33da144aed6347ddbef4eca13883aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.897814 2010 tablet_bootstrap.cc:492] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: Bootstrap starting.
I20260925 14:49:05.897827 2015 tablet_bootstrap.cc:492] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: Bootstrap starting.
I20260925 14:49:05.897914 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 33da144aed6347ddbef4eca13883aabb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=576d9c0fc616435386b9001151311541]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:05.898053 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33da144aed6347ddbef4eca13883aabb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:05.898237 2010 tablet_bootstrap.cc:654] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.898244 2015 tablet_bootstrap.cc:654] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.898887 2010 tablet_bootstrap.cc:492] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: No bootstrap required, opened a new log
I20260925 14:49:05.898928 2010 ts_tablet_manager.cc:1397] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:05.899060 2010 raft_consensus.cc:348] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.899127 2010 raft_consensus.cc:374] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.899144 2010 raft_consensus.cc:729] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a860375bcec4542800dd071bd96e183, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.899180 2010 consensus_queue.cc:260] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.899253 2010 ts_tablet_manager.cc:1428] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.899367 2014 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:05.899435 2014 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:05.899636 2011 tablet_bootstrap.cc:492] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: Bootstrap starting.
I20260925 14:49:05.899797 1490 catalog_manager.cc:4841] TS 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639): tablet 96026b1c94ca4203a056e9dce1c6575d (table test-table [id=98bc947e547242169143f3b77288a921]) successfully deleted
I20260925 14:49:05.900502 2011 tablet_bootstrap.cc:654] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:05.901087 2015 tablet_bootstrap.cc:492] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: No bootstrap required, opened a new log
I20260925 14:49:05.901125 2015 ts_tablet_manager.cc:1397] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:49:05.901234 2015 raft_consensus.cc:348] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.901283 2015 raft_consensus.cc:374] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.901295 2015 raft_consensus.cc:729] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6481f64f2ec0440792a4c5dc068d68a7, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.901331 2015 consensus_queue.cc:260] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.901429 2015 ts_tablet_manager.cc:1428] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:05.901458 2030 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:05.901537 2030 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:05.902047 2011 tablet_bootstrap.cc:492] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: No bootstrap required, opened a new log
I20260925 14:49:05.902096 2011 ts_tablet_manager.cc:1397] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:49:05.902237 2011 raft_consensus.cc:348] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.902303 2011 raft_consensus.cc:374] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:05.902355 2011 raft_consensus.cc:729] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c1268bbffac4f138d8afe23c5c5cc0f, State: Initialized, Role: FOLLOWER
I20260925 14:49:05.902566 2011 consensus_queue.cc:260] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:05.902804 2011 ts_tablet_manager.cc:1428] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:05.902941 2016 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:05.903025 2016 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:05.914921 1488 catalog_manager.cc:4841] TS 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147): tablet 96026b1c94ca4203a056e9dce1c6575d (table test-table [id=98bc947e547242169143f3b77288a921]) successfully deleted
I20260925 14:49:06.076097 2014 raft_consensus.cc:482] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:06.076210 2014 raft_consensus.cc:504] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:06.076596 2014 leader_election.cc:288] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147), 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3:42041)
I20260925 14:49:06.079674 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.076918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40543 (local address 127.0.98.2:39147)
0925 14:49:06.076971 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.076974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.077086 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:49:06.077148 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.077150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.077159 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.077216 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:06.077375 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.077871 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.078955 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.079316 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.079372 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.079379 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.079559 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.079603 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:49:06.079710 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.076924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55781 (local address 127.0.98.3:42041)
0925 14:49:06.077006 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.077009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.077130 (+ 121us) server_negotiation.cc:407] Connection header received
0925 14:49:06.077169 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.077169 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.077178 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.077256 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:06.077444 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.077887 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.079011 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.079386 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.079456 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.079465 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.079610 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.079648 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:49:06.079715 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.076790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39147 (local address 127.0.98.1:40543)
0925 14:49:06.077003 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:06.077015 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:06.077087 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:06.077239 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:06.077240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:06.077243 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:06.077306 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.077308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.077956 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.077957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.078874 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.078876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.079396 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.079415 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.079487 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.079492 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:06.079549 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:49:06.079610 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":138,"threads_started":1}
I20260925 14:49:06.079733 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.076792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42041 (local address 127.0.98.1:55781)
0925 14:49:06.077022 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:06.077032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:06.077119 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:06.077272 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:06.077272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:06.077274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:06.077334 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.077335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.078022 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.078023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.078900 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.078903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.079471 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.079472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.079515 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.079518 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:06.079591 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:49:06.079629 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":67,"thread_start_us":109,"threads_started":1}
I20260925 14:49:06.080160 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da144aed6347ddbef4eca13883aabb" candidate_uuid: "2a860375bcec4542800dd071bd96e183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" is_pre_election: true
I20260925 14:49:06.080170 1930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da144aed6347ddbef4eca13883aabb" candidate_uuid: "2a860375bcec4542800dd071bd96e183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6481f64f2ec0440792a4c5dc068d68a7" is_pre_election: true
I20260925 14:49:06.080259 1801 raft_consensus.cc:2454] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a860375bcec4542800dd071bd96e183 in term 0.
I20260925 14:49:06.080287 1930 raft_consensus.cc:2454] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a860375bcec4542800dd071bd96e183 in term 0.
I20260925 14:49:06.080451 1592 leader_election.cc:302] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a860375bcec4542800dd071bd96e183, 9c1268bbffac4f138d8afe23c5c5cc0f; no voters:
I20260925 14:49:06.080608 2014 raft_consensus.cc:2781] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:06.080657 2014 raft_consensus.cc:482] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:06.080689 2014 raft_consensus.cc:3037] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:06.081380 2014 raft_consensus.cc:504] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:06.081524 2014 leader_election.cc:288] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [CANDIDATE]: Term 1 election: Requested vote from peers 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147), 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3:42041)
I20260925 14:49:06.081642 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da144aed6347ddbef4eca13883aabb" candidate_uuid: "2a860375bcec4542800dd071bd96e183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f"
I20260925 14:49:06.081650 1930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da144aed6347ddbef4eca13883aabb" candidate_uuid: "2a860375bcec4542800dd071bd96e183" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6481f64f2ec0440792a4c5dc068d68a7"
I20260925 14:49:06.081717 1801 raft_consensus.cc:3037] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:06.081727 1930 raft_consensus.cc:3037] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:06.082490 1801 raft_consensus.cc:2454] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a860375bcec4542800dd071bd96e183 in term 1.
I20260925 14:49:06.082618 1930 raft_consensus.cc:2454] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a860375bcec4542800dd071bd96e183 in term 1.
I20260925 14:49:06.082700 1592 leader_election.cc:302] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a860375bcec4542800dd071bd96e183, 9c1268bbffac4f138d8afe23c5c5cc0f; no voters:
I20260925 14:49:06.082805 2014 raft_consensus.cc:2781] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:06.082973 2014 raft_consensus.cc:686] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [term 1 LEADER]: Becoming Leader. State: Replica: 2a860375bcec4542800dd071bd96e183, State: Running, Role: LEADER
I20260925 14:49:06.083056 2014 consensus_queue.cc:237] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } }
I20260925 14:49:06.083407 2052 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: TxnStatusTablet state changed. Reason: New leader 2a860375bcec4542800dd071bd96e183. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.083483 2052 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:06.083680 2054 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:49:06.083801 2054 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:49:06.083812 1504 catalog_manager.cc:5495] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a860375bcec4542800dd071bd96e183 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:06.131542 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.129043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46988 (local address 127.0.98.1:33639)
0925 14:49:06.129097 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.129100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.129112 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:49:06.129150 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.129151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.129179 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.129240 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:06.129390 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.129867 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.130723 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.130933 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.131013 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.131189 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:06.131296 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:06.131380 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.131431 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.131458 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:49:06.131546 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.128933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33639 (local address 127.0.0.1:46988)
0925 14:49:06.129014 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:06.129027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:06.129078 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:06.129263 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:06.129264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:06.129266 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:06.129315 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.129317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.129979 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.129980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.130637 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.130639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.131011 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.131012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.131070 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.131086 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:06.131367 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:06.131376 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:49:06.131427 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:49:06.131478 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:49:06.132754 1801 raft_consensus.cc:1261] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Refusing update from remote peer 2a860375bcec4542800dd071bd96e183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:06.132804 1930 raft_consensus.cc:1261] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Refusing update from remote peer 2a860375bcec4542800dd071bd96e183: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:06.133036 2014 consensus_queue.cc:1035] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:06.133147 2052 consensus_queue.cc:1035] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:06.134428 2016 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: TxnStatusTablet state changed. Reason: New leader 2a860375bcec4542800dd071bd96e183. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.134532 2016 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:06.134940 2014 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.135030 2014 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:06.135042 2030 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: TxnStatusTablet state changed. Reason: New leader 2a860375bcec4542800dd071bd96e183. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.135143 2030 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:06.136288 2030 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.136384 2030 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 6481f64f2ec0440792a4c5dc068d68a7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:06.136698 2014 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.136775 2014 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 2a860375bcec4542800dd071bd96e183: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:06.137180 2016 tablet_replica.cc:439] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a860375bcec4542800dd071bd96e183" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 } } peers { permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 } } peers { permanent_uuid: "6481f64f2ec0440792a4c5dc068d68a7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42041 } } }
I20260925 14:49:06.137266 2016 tablet_replica.cc:442] T 33da144aed6347ddbef4eca13883aabb P 9c1268bbffac4f138d8afe23c5c5cc0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:06.142982 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.139457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36073 (local address 127.0.98.1:33639)
0925 14:49:06.139497 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.139500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.139989 (+ 489us) server_negotiation.cc:407] Connection header received
0925 14:49:06.140040 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.140041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.140054 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.140122 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:06.140282 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.140952 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.142046 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.142362 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.142497 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.142685 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:06.142724 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:06.142786 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.142872 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.142901 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:06.143049 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.139321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33639 (local address 127.0.98.3:36073)
0925 14:49:06.139477 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:06.139492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:06.139971 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:06.140140 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:06.140141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:06.140143 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:06.140199 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.140201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.141055 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.141056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.141938 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.141941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.142499 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.142501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.142564 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.142585 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:06.142807 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:06.142810 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:06.142867 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:49:06.142962 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260925 14:49:06.145419 1801 raft_consensus.cc:1261] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Refusing update from remote peer 6481f64f2ec0440792a4c5dc068d68a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:06.145476 1661 raft_consensus.cc:1261] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Refusing update from remote peer 6481f64f2ec0440792a4c5dc068d68a7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:06.145639 2018 consensus_queue.cc:1035] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c1268bbffac4f138d8afe23c5c5cc0f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:06.145892 2018 consensus_queue.cc:1035] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a860375bcec4542800dd071bd96e183" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:06.154996 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.151967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42925 (local address 127.0.98.3:42041)
0925 14:49:06.152011 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.152013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.152152 (+ 139us) server_negotiation.cc:407] Connection header received
0925 14:49:06.152184 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.152184 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.152197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.152268 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:06.152369 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.153135 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.154261 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.154521 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.154578 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.154772 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:06.154801 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:06.154838 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.154893 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.154912 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:06.155049 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.151914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42041 (local address 127.0.98.1:42925)
0925 14:49:06.152075 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:06.152085 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:06.152148 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:06.152268 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:06.152269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:06.152270 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:06.152323 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.152324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.153233 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.153235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.154177 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:06.154180 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.154597 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.154598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:06.154671 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.154689 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:06.154840 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:06.154842 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:06.154892 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:49:06.154937 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260925 14:49:06.163582 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:06.163707 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:06.164350 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 186a79c77bef4291856867d22d1ff4ab: Sending DeleteTablet for 3 replicas of tablet 79289343173348abaa8f4c6026d5cf86
I20260925 14:49:06.164597 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 79289343173348abaa8f4c6026d5cf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46982
I20260925 14:49:06.164659 1920 tablet_service.cc:1514] Processing DeleteTablet for tablet 79289343173348abaa8f4c6026d5cf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44964
I20260925 14:49:06.164695 2033 tablet_replica.cc:330] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: stopping tablet replica
I20260925 14:49:06.164734 2035 tablet_replica.cc:330] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: stopping tablet replica
I20260925 14:49:06.164738 2033 raft_consensus.cc:2229] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:06.164772 2033 raft_consensus.cc:2258] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:06.164777 2035 raft_consensus.cc:2229] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:06.164829 2035 raft_consensus.cc:2258] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:06.164947 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 79289343173348abaa8f4c6026d5cf86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52124
I20260925 14:49:06.165035 2034 tablet_replica.cc:330] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: stopping tablet replica
I20260925 14:49:06.165084 2034 raft_consensus.cc:2229] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:06.165122 2034 raft_consensus.cc:2258] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:06.165242 2035 ts_tablet_manager.cc:1905] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:06.165503 2033 ts_tablet_manager.cc:1905] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:06.166669 2034 ts_tablet_manager.cc:1905] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:06.167172 2033 ts_tablet_manager.cc:1918] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:06.167238 2033 log.cc:1196] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/79289343173348abaa8f4c6026d5cf86
I20260925 14:49:06.167475 2033 ts_tablet_manager.cc:1939] T 79289343173348abaa8f4c6026d5cf86 P 2a860375bcec4542800dd071bd96e183: Deleting consensus metadata
I20260925 14:49:06.167986 1490 catalog_manager.cc:4841] TS 2a860375bcec4542800dd071bd96e183 (127.0.98.1:33639): tablet 79289343173348abaa8f4c6026d5cf86 (table test-table [id=458817f3cded4728aaf145acf6ccdec3]) successfully deleted
I20260925 14:49:06.168241 2034 ts_tablet_manager.cc:1918] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:06.168313 2034 log.cc:1196] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/79289343173348abaa8f4c6026d5cf86
I20260925 14:49:06.168538 2034 ts_tablet_manager.cc:1939] T 79289343173348abaa8f4c6026d5cf86 P 9c1268bbffac4f138d8afe23c5c5cc0f: Deleting consensus metadata
I20260925 14:49:06.168789 2035 ts_tablet_manager.cc:1918] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:06.168849 2035 log.cc:1196] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/79289343173348abaa8f4c6026d5cf86
I20260925 14:49:06.169034 2035 ts_tablet_manager.cc:1939] T 79289343173348abaa8f4c6026d5cf86 P 6481f64f2ec0440792a4c5dc068d68a7: Deleting consensus metadata
I20260925 14:49:06.169329 1488 catalog_manager.cc:4841] TS 9c1268bbffac4f138d8afe23c5c5cc0f (127.0.98.2:39147): tablet 79289343173348abaa8f4c6026d5cf86 (table test-table [id=458817f3cded4728aaf145acf6ccdec3]) successfully deleted
I20260925 14:49:06.169360 1491 catalog_manager.cc:4841] TS 6481f64f2ec0440792a4c5dc068d68a7 (127.0.98.3:42041): tablet 79289343173348abaa8f4c6026d5cf86 (table test-table [id=458817f3cded4728aaf145acf6ccdec3]) successfully deleted
Sep 25 14:49:06 dist-test-slave-z2s5 krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:06.177891 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.168678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.0.98.1:33639)
0925 14:49:06.168746 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.168748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.169441 (+ 693us) server_negotiation.cc:407] Connection header received
0925 14:49:06.169514 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.169515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.169525 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.169574 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:06.170144 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.170679 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.171416 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.171604 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.171691 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.174775 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:06.174783 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:06.174784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:06.174791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:06.176620 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:06.176991 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:06.176994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:06.176994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:06.177065 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:06.177300 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:06.177301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:06.177301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:06.177471 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:06.177606 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.177680 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.177761 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 14:49:06.178176 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46992
I20260925 14:49:06.185938 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:06.179163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57856 (local address 127.0.98.62:41511)
0925 14:49:06.179207 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:06.179208 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:06.179218 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:06.179261 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:06.179262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:06.179275 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:06.179353 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:06.179806 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.180582 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.181283 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:06.181586 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:06.181669 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:06.183042 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:06.183043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:06.183044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:06.183052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:06.184661 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:06.185037 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:06.185038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:06.185038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:06.185106 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:06.185366 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:06.185367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:06.185367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:06.185505 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:06.185617 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:06.185731 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:49:06.185835 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:49:06.186146 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57856
I20260925 14:49:06.186820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1576
I20260925 14:49:06.193027 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1716
I20260925 14:49:06.197757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1856
I20260925 14:49:06.203306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 1472
2026-09-25T14:49:06Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2717 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:06 dist-test-slave-z2s5 krb5kdc[2073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:06 dist-test-slave-z2s5 krb5kdc[2073](info): set up 2 sockets
Sep 25 14:49:06 dist-test-slave-z2s5 krb5kdc[2073](info): commencing operation
krb5kdc: starting...
W20260925 14:49:07.258090 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:07.375262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42275
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:07.454339 2086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:07.454612 2086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:07.454650 2086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:07.455782 2086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:07.455868 2086 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:49:07.455890 2086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:07.455914 2086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:07.455941 2086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:07.455960 2086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:07.457391 2086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42275
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:07.457669 2086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:07.457903 2086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:07.462954 2091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:07.463131 2092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:07.463202 2094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:07.463428 2086 server_base.cc:1034] running on GCE node
I20260925 14:49:07.463681 2086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:07.463975 2086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:07.465147 2086 hybrid_clock.cc:648] HybridClock initialized: now 1790347747465119 us; error 45 us; skew 500 ppm
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:07.467911 2086 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:07.468384 2086 webserver.cc:460] Webserver started at http://127.0.98.62:38697/ using document root <none> and password file <none>
I20260925 14:49:07.468628 2086 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:07.468698 2086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:07.468887 2086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:07.469928 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "2a08fdd0d4dd4d038745ac9250a8cced"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "b5e7b5713a5dd59cda2c16ccd7826a33"
server_key_iv: "be2c61198c641bdebbb7de1573d54749"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.470285 2086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "2a08fdd0d4dd4d038745ac9250a8cced"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "b5e7b5713a5dd59cda2c16ccd7826a33"
server_key_iv: "be2c61198c641bdebbb7de1573d54749"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.471614 2086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:07.472380 2101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.472554 2086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.472643 2086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "2a08fdd0d4dd4d038745ac9250a8cced"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "b5e7b5713a5dd59cda2c16ccd7826a33"
server_key_iv: "be2c61198c641bdebbb7de1573d54749"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.472728 2086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:07.495836 2086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:07.498271 2086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:07.498472 2086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:07.503103 2086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42275
I20260925 14:49:07.503150 2163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42275 every 8 connection(s)
I20260925 14:49:07.503464 2086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:07.504065 2164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:07.506297 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Bootstrap starting.
I20260925 14:49:07.506910 2164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:07.507200 2164 log.cc:824] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:07.507874 2164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: No bootstrap required, opened a new log
I20260925 14:49:07.510037 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 2086
I20260925 14:49:07.510130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:07.510054 2164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42275 } }
I20260925 14:49:07.510236 2164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:07.510272 2164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a08fdd0d4dd4d038745ac9250a8cced, State: Initialized, Role: FOLLOWER
I20260925 14:49:07.510294 392 external_mini_cluster.cc:1428] Setting key 9fcd9f5b1077ffb6f0063ce6fda84019
I20260925 14:49:07.510449 2164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42275 } }
I20260925 14:49:07.510562 2164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:07.510607 2164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:07.510658 2164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:07.511345 2164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42275 } }
I20260925 14:49:07.511471 2164 leader_election.cc:302] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a08fdd0d4dd4d038745ac9250a8cced; no voters:
I20260925 14:49:07.511628 2164 leader_election.cc:288] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:07.511652 2169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:07.511816 2169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [term 1 LEADER]: Becoming Leader. State: Replica: 2a08fdd0d4dd4d038745ac9250a8cced, State: Running, Role: LEADER
I20260925 14:49:07.511883 2164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:07.511909 2169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42275 } }
I20260925 14:49:07.512250 2169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a08fdd0d4dd4d038745ac9250a8cced. Latest consensus state: current_term: 1 leader_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42275 } } }
I20260925 14:49:07.512338 2169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:07.512260 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a08fdd0d4dd4d038745ac9250a8cced" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42275 } } }
I20260925 14:49:07.512509 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:07.512522 2176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:07.513078 2176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:07.515033 2176 catalog_manager.cc:1347] Generated new cluster ID: 31320da9d5834619a16bb224f8d1b062
I20260925 14:49:07.515084 2176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:07.520550 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.510782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46632 (local address 127.0.98.62:42275)
0925 14:49:07.510961 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:07.510970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:07.511012 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:49:07.511106 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:07.511112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:07.511153 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:07.511263 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:07.512099 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.513558 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.514504 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.515054 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.515175 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.517422 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:07.517426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:07.517426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:07.517450 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:07.519061 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.519439 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.519442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.519442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.519545 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.519754 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.519755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.519755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.519983 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:07.520164 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:07.520228 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:49:07.520363 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260925 14:49:07.533730 2176 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:07.534238 2176 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:07.537674 2176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Generated new TSK 0
I20260925 14:49:07.537850 2176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:07.569551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42275
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:07.668610 2190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:07.668849 2190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:07.668889 2190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:07.670049 2190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:07.670116 2190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:07.670156 2190 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:49:07.670181 2190 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:07.670218 2190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:07.671691 2190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:07.671962 2190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:07.672196 2190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:07.677275 2196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:07.677311 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:07.677301 2190 server_base.cc:1034] running on GCE node
W20260925 14:49:07.677242 2198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:07.677616 2190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:07.677858 2190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:07.679014 2190 hybrid_clock.cc:648] HybridClock initialized: now 1790347747679001 us; error 45 us; skew 500 ppm
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:07.688527 2190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:07.689013 2190 webserver.cc:460] Webserver started at http://127.0.98.1:41121/ using document root <none> and password file <none>
I20260925 14:49:07.689265 2190 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:07.689332 2190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:07.689476 2190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:07.690604 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "245397fad9f84c9db25cfdffd719083d"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "a23a37c880ac828c08b15f7b7895c2b3"
server_key_iv: "21f77fb34453d2e328e3f4695375baf8"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.690969 2190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "245397fad9f84c9db25cfdffd719083d"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "a23a37c880ac828c08b15f7b7895c2b3"
server_key_iv: "21f77fb34453d2e328e3f4695375baf8"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.692286 2190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:07.692981 2205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.693145 2190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:07.693241 2190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "245397fad9f84c9db25cfdffd719083d"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "a23a37c880ac828c08b15f7b7895c2b3"
server_key_iv: "21f77fb34453d2e328e3f4695375baf8"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.693320 2190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:07.714845 2190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:07.716662 2190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:07.716825 2190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:07.717366 2190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:07.717435 2190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.717469 2190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:07.717504 2190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:07.724841 2190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33999
I20260925 14:49:07.724998 2320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33999 every 8 connection(s)
I20260925 14:49:07.725262 2190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:07.727391 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.717801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53901 (local address 127.0.98.62:42275)
0925 14:49:07.717872 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:07.717875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:07.718122 (+ 247us) server_negotiation.cc:407] Connection header received
0925 14:49:07.718678 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:07.718680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:07.718698 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:07.718777 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:07.719362 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.720197 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.721273 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.721480 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.721568 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.724323 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:07.724324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:07.724325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:07.724337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:07.725959 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.726412 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.726414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.726414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.726476 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.726724 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.726725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.726726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.726849 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:07.726926 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:07.727217 (+ 291us) server_negotiation.cc:299] Negotiation successful
0925 14:49:07.727302 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:49:07.727592 2214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.717783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.98.1:53901)
0925 14:49:07.717963 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:07.718024 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:07.718571 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:07.718859 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:07.718863 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:07.719109 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:07.719284 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.719289 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.720314 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.720315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.721176 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.721179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.721577 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.721578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.721708 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.722293 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:07.722303 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:07.724217 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:07.726067 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.726074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.726076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.726300 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.726543 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.726544 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.726544 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.726654 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.726916 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:07.726922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:07.727081 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 14:49:07.727194 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:49:07.727355 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":116,"threads_started":1}
I20260925 14:49:07.728093 2321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42275
I20260925 14:49:07.728195 2321 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:07.728389 2321 heartbeater.cc:511] Master 127.0.98.62:42275 requested a full tablet report, sending...
I20260925 14:49:07.728945 2117 ts_manager.cc:194] Registered new tserver with Master: 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999)
I20260925 14:49:07.729873 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53901
I20260925 14:49:07.734325 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.729225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40215 (local address 127.0.98.62:42275)
0925 14:49:07.729264 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:07.729266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:07.729396 (+ 130us) server_negotiation.cc:407] Connection header received
0925 14:49:07.729464 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:07.729464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:07.729473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:07.729513 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:07.729823 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.730608 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.731276 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.731457 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.731510 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.732363 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:07.732363 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:07.732363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:07.732368 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:07.733452 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.733738 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.733739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.733739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.733784 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.733975 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.733976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.733976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.734064 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:07.734114 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:07.734216 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:49:07.734270 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":19}
I20260925 14:49:07.734340 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.729140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.98.1:40215)
0925 14:49:07.729323 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:07.729334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:07.729408 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:07.729525 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:07.729526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:07.729701 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:07.729760 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.729761 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.730693 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.730694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.731205 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.731206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.731524 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.731525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.731579 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.732027 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:07.732032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:07.732295 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:07.733523 (+ 1228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.733524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.733524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.733677 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.733866 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.733866 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.733867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.733914 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.734119 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:07.734120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:07.734152 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:49:07.734203 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:49:07.734274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
I20260925 14:49:07.735039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 2190
I20260925 14:49:07.735116 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:07.735328 392 external_mini_cluster.cc:1428] Setting key 88101de2aa86a8a6229b755152bfe899
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:07.760543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42275
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:07.840519 2330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:07.840728 2330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:07.840752 2330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:07.841928 2330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:07.841974 2330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:07.842001 2330 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:49:07.842013 2330 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:07.842082 2330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:07.843580 2330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:07.843835 2330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:07.844069 2330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:07.849112 2335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:07.849124 2336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:07.849131 2338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:07.849490 2330 server_base.cc:1034] running on GCE node
I20260925 14:49:07.849704 2330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:07.849972 2330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:07.851155 2330 hybrid_clock.cc:648] HybridClock initialized: now 1790347747851113 us; error 64 us; skew 500 ppm
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:07.853726 2330 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:07.854198 2330 webserver.cc:460] Webserver started at http://127.0.98.2:36119/ using document root <none> and password file <none>
I20260925 14:49:07.854485 2330 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:07.854533 2330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:07.854671 2330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:07.855743 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "7d7ac715ab954fae9c0db65f646386b0"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "933d5df2b477b619436d355a5587a67b"
server_key_iv: "f77cdd38dd2015c0a981bf78e4089132"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.856117 2330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7d7ac715ab954fae9c0db65f646386b0"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "933d5df2b477b619436d355a5587a67b"
server_key_iv: "f77cdd38dd2015c0a981bf78e4089132"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.857548 2330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:07.858362 2345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.858621 2330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:07.858719 2330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "7d7ac715ab954fae9c0db65f646386b0"
format_stamp: "Formatted at 2026-09-25 14:49:07 on dist-test-slave-z2s5"
server_key: "933d5df2b477b619436d355a5587a67b"
server_key_iv: "f77cdd38dd2015c0a981bf78e4089132"
server_key_version: "encryptionkey@0"
I20260925 14:49:07.858807 2330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:07.875177 2330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:07.877678 2330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:07.877846 2330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:07.878479 2330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:07.878532 2330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.878566 2330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:07.878585 2330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:07.885879 2330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33491
I20260925 14:49:07.885984 2460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33491 every 8 connection(s)
Sep 25 14:49:07 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:07.886309 2330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:07.889158 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.878913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52505 (local address 127.0.98.62:42275)
0925 14:49:07.878976 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:07.878978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:07.879264 (+ 286us) server_negotiation.cc:407] Connection header received
0925 14:49:07.880020 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:07.880021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:07.880034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:07.880097 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:07.881022 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.881582 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.883259 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.883462 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.883513 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.886876 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:07.886877 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:07.886877 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:07.886883 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:07.888059 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.888402 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.888403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.888403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.888449 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.888661 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.888662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.888662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.888761 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:07.888810 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:07.889033 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 14:49:07.889094 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:49:07.889345 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.878911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.98.2:52505)
0925 14:49:07.879101 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:07.879158 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:07.879886 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:07.880181 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:07.880186 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:07.880588 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:07.880915 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.880923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.881656 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.881657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.883145 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.883149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.883537 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.883538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.883710 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.884822 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:07.884834 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:07.886784 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:07.888126 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.888132 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.888133 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.888339 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.888517 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.888518 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.888518 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.888609 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.888813 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:07.888817 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:07.888926 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 14:49:07.889010 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:49:07.889126 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
I20260925 14:49:07.889834 2461 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42275
I20260925 14:49:07.889932 2461 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:07.890134 2461 heartbeater.cc:511] Master 127.0.98.62:42275 requested a full tablet report, sending...
I20260925 14:49:07.890563 2118 ts_manager.cc:194] Registered new tserver with Master: 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491)
I20260925 14:49:07.890928 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52505
I20260925 14:49:07.895532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 2330
I20260925 14:49:07.895624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:07.895833 392 external_mini_cluster.cc:1428] Setting key b91777d89e5d9c3369471f707fad8c51
I20260925 14:49:07.896206 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.890948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.98.2:54379)
0925 14:49:07.891089 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:07.891100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:07.891170 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:07.891316 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:07.891317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:07.891462 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:07.891516 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.891517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.892174 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.892174 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.892668 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:07.892669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.892961 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.892962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:07.893015 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.893369 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:07.893372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:07.893639 (+ 267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:07.895146 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.895146 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.895147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.895302 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.895527 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:07.895528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:07.895528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:07.895576 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:07.895955 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:07.895956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:07.896001 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:49:07.896060 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:49:07.896135 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260925 14:49:07.896268 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:07.891007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54379 (local address 127.0.98.62:42275)
0925 14:49:07.891051 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:07.891053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:07.891145 (+ 92us) server_negotiation.cc:407] Connection header received
0925 14:49:07.891242 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:07.891243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:07.891253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:07.891308 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:07.891587 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.892085 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.892734 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:07.892903 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:07.892949 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:07.893710 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:07.893711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:07.893711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:07.893716 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:07.895079 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.895374 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.895375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.895375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.895436 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:07.895667 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:07.895668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:07.895669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:07.895892 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:07.895966 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:07.896109 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:49:07.896184 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:07.921015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42275
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:08.001541 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:08.001748 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:08.001775 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:08.002872 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:08.002908 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:08.002929 2470 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:49:08.002947 2470 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:08.002965 2470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:08.004351 2470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:08.004539 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:08.004707 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:08.009802 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:08.009793 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:08.009824 2470 server_base.cc:1034] running on GCE node
W20260925 14:49:08.009793 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:08.010294 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:08.010567 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:08.011732 2470 hybrid_clock.cc:648] HybridClock initialized: now 1790347748011699 us; error 51 us; skew 500 ppm
Sep 25 14:49:08 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:08.014451 2470 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:08.014914 2470 webserver.cc:460] Webserver started at http://127.0.98.3:40625/ using document root <none> and password file <none>
I20260925 14:49:08.015152 2470 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:08.015219 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:08.015349 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:08.016409 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c09721b37cb4f76b8ef919060930ae0"
format_stamp: "Formatted at 2026-09-25 14:49:08 on dist-test-slave-z2s5"
server_key: "d8a3977f83caeeb37e25111ef9ed540a"
server_key_iv: "856bc7b91a01d30e4322dd808eaf3cb8"
server_key_version: "encryptionkey@0"
I20260925 14:49:08.016764 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c09721b37cb4f76b8ef919060930ae0"
format_stamp: "Formatted at 2026-09-25 14:49:08 on dist-test-slave-z2s5"
server_key: "d8a3977f83caeeb37e25111ef9ed540a"
server_key_iv: "856bc7b91a01d30e4322dd808eaf3cb8"
server_key_version: "encryptionkey@0"
I20260925 14:49:08.018082 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:08.018865 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:08.019063 2470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:08.019160 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "8c09721b37cb4f76b8ef919060930ae0"
format_stamp: "Formatted at 2026-09-25 14:49:08 on dist-test-slave-z2s5"
server_key: "d8a3977f83caeeb37e25111ef9ed540a"
server_key_iv: "856bc7b91a01d30e4322dd808eaf3cb8"
server_key_version: "encryptionkey@0"
I20260925 14:49:08.019237 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:08.036682 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:08.038878 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:08.039047 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:08.039594 2470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:08.039651 2470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:08.039695 2470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:08.039719 2470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:49:08 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:08.046840 2470 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36545
I20260925 14:49:08.046923 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36545 every 8 connection(s)
I20260925 14:49:08.047281 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:08.049422 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.040039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58611 (local address 127.0.98.62:42275)
0925 14:49:08.040105 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.040108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.040336 (+ 228us) server_negotiation.cc:407] Connection header received
0925 14:49:08.040818 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.040819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.040833 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.040920 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:08.041584 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.042431 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.043527 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.043880 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.043968 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.046853 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:08.046854 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:08.046855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:08.046863 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:08.048310 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.048671 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.048672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.048672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.048720 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.048921 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.048922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.048922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.049027 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:08.049078 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.049297 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.049360 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:49:08.049631 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.040019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.98.3:58611)
0925 14:49:08.040206 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.040246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.040728 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.040975 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.040978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.041277 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:08.041474 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.041479 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.042551 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.042552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.043459 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.043462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.044048 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.044049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.044217 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.044796 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:08.044811 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:08.046755 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:08.048381 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:08.048385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:08.048387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:08.048607 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:08.048776 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:08.048777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:08.048777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:08.048870 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:08.049081 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:08.049085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:08.049196 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.049275 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.049390 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260925 14:49:08.050197 2601 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42275
I20260925 14:49:08.050277 2601 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:08.050479 2601 heartbeater.cc:511] Master 127.0.98.62:42275 requested a full tablet report, sending...
I20260925 14:49:08.050848 2117 ts_manager.cc:194] Registered new tserver with Master: 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:08.051257 2117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58611
I20260925 14:49:08.056134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 2470
I20260925 14:49:08.056227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:08.056311 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.051341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49447 (local address 127.0.98.62:42275)
0925 14:49:08.051380 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.051381 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.051522 (+ 141us) server_negotiation.cc:407] Connection header received
0925 14:49:08.051598 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.051598 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.051608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.051674 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:08.051945 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.052443 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.053096 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.053269 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.053320 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.054082 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:08.054083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:08.054083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:08.054089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:08.055315 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.055615 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.055615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.055616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.055662 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.055815 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.055816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.055816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.055905 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:08.055970 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.056129 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.056181 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:08.056336 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.051253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.98.3:49447)
0925 14:49:08.051461 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.051473 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.051550 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.051678 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.051679 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.051821 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:08.051875 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.051877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.052533 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.052534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.053028 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.053029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.053332 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.053332 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.053376 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.053744 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:08.053748 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:08.054011 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:08.055381 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:08.055382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:08.055382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:08.055552 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:08.055717 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:08.055717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:08.055717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:08.055765 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:08.055982 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:08.055983 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:08.056032 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.056114 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.056224 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":86,"threads_started":1}
I20260925 14:49:08.056406 392 external_mini_cluster.cc:1428] Setting key f289bd55a9e0c499540f3b34d3c77e20
I20260925 14:49:08.057541 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:49:08 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:49:08 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:08.080669 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.073602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.0.98.62:42275)
0925 14:49:08.073669 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.073671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.073686 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:49:08.073712 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.073712 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.073736 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.073788 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:08.074361 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.074965 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.075752 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.076004 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.076099 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.078249 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:08.078250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:08.078250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:08.078256 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:08.079699 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.079992 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.079993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.079993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.080041 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.080196 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.080197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.080197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.080301 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:08.080375 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.080548 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.080606 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:49:08.082525 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:08.084117 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:08.090617 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.086546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.0.1:52770)
0925 14:49:08.086912 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.086968 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.087016 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.087434 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.087435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.087437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.087485 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.087486 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.088322 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.088323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.089376 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.089378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.090270 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.090271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.090346 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.090359 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.090478 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.090534 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":1,"thread_start_us":56,"threads_started":1}
I20260925 14:49:08.090631 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.086704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52770 (local address 127.0.98.1:33999)
0925 14:49:08.086950 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.086956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.087227 (+ 271us) server_negotiation.cc:407] Connection header received
0925 14:49:08.087277 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.087279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.087295 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.087366 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.087541 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.088235 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.089471 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.090189 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.090272 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.090333 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.090504 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.090541 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":98,"threads_started":1}
I20260925 14:49:08.091519 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.086785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39304 (local address 127.0.98.3:36545)
0925 14:49:08.086987 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.086991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.087169 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:49:08.087213 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.087215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.087232 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.087307 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.087504 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.088756 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.090117 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.091053 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.091115 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.091165 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.091381 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.091421 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260925 14:49:08.091619 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.086721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52050 (local address 127.0.98.2:33491)
0925 14:49:08.087042 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.087046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.087081 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:49:08.087117 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.087119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.087158 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.087243 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.087506 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.088754 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.090193 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.091025 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.091105 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.091165 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.091464 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.091509 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":78,"threads_started":1}
I20260925 14:49:08.091655 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 91b8d19188ea417b97261c283e43b94d (DEFAULT_TABLE table=test-table [id=24f54c7707064f3ca7d14f796ead172f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:08.091813 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.086548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.0.1:39304)
0925 14:49:08.087090 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.087098 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.087142 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.087378 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.087379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.087381 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.087414 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.087416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.088863 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.088864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.090030 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.090032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.091204 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.091205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.091287 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.091290 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.091364 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.091395 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":30,"spinlock_wait_cycles":896,"thread_start_us":101,"threads_started":1}
I20260925 14:49:08.091956 2618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.086543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33491 (local address 127.0.0.1:52050)
0925 14:49:08.086890 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.086933 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.087004 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.087327 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.087331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.087333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.087414 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.087418 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.088875 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.088876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.090088 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.090091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.091205 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.091205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.091285 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.091289 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.091346 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.091390 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":23,"thread_start_us":90,"threads_started":1}
I20260925 14:49:08.092048 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b8d19188ea417b97261c283e43b94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.092569 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 91b8d19188ea417b97261c283e43b94d (DEFAULT_TABLE table=test-table [id=24f54c7707064f3ca7d14f796ead172f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:08.092708 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 91b8d19188ea417b97261c283e43b94d (DEFAULT_TABLE table=test-table [id=24f54c7707064f3ca7d14f796ead172f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:08.092917 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b8d19188ea417b97261c283e43b94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.093025 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b8d19188ea417b97261c283e43b94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.094919 2624 tablet_bootstrap.cc:492] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Bootstrap starting.
I20260925 14:49:08.095575 2624 tablet_bootstrap.cc:654] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.095834 2624 log.cc:824] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:08.096563 2626 tablet_bootstrap.cc:492] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Bootstrap starting.
I20260925 14:49:08.096684 2624 tablet_bootstrap.cc:492] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: No bootstrap required, opened a new log
I20260925 14:49:08.096743 2624 ts_tablet_manager.cc:1397] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:08.096794 2627 tablet_bootstrap.cc:492] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Bootstrap starting.
I20260925 14:49:08.097375 2627 tablet_bootstrap.cc:654] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.097487 2626 tablet_bootstrap.cc:654] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.097754 2627 log.cc:824] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:08.097815 2626 log.cc:824] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:08.098646 2627 tablet_bootstrap.cc:492] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: No bootstrap required, opened a new log
I20260925 14:49:08.098718 2627 ts_tablet_manager.cc:1397] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:08.099041 2624 raft_consensus.cc:348] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.099227 2624 raft_consensus.cc:374] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.099260 2624 raft_consensus.cc:729] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245397fad9f84c9db25cfdffd719083d, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.099390 2624 consensus_queue.cc:260] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.099666 2624 ts_tablet_manager.cc:1428] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:08.099709 2321 heartbeater.cc:503] Master 127.0.98.62:42275 was elected leader, sending a full tablet report...
I20260925 14:49:08.100739 2627 raft_consensus.cc:348] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.100898 2627 raft_consensus.cc:374] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.100934 2627 raft_consensus.cc:729] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.101073 2627 consensus_queue.cc:260] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.101285 2627 ts_tablet_manager.cc:1428] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:49:08.101349 2461 heartbeater.cc:503] Master 127.0.98.62:42275 was elected leader, sending a full tablet report...
I20260925 14:49:08.101724 2626 tablet_bootstrap.cc:492] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: No bootstrap required, opened a new log
I20260925 14:49:08.101795 2626 ts_tablet_manager.cc:1397] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:49:08.103351 2626 raft_consensus.cc:348] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.103477 2626 raft_consensus.cc:374] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.103498 2626 raft_consensus.cc:729] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.103652 2626 consensus_queue.cc:260] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.103919 2626 ts_tablet_manager.cc:1428] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:08.103972 2601 heartbeater.cc:503] Master 127.0.98.62:42275 was elected leader, sending a full tablet report...
W20260925 14:49:08.136912 2462 tablet.cc:2369] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:08.186697 2631 raft_consensus.cc:482] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:08.186825 2631 raft_consensus.cc:504] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.187165 2631 leader_election.cc:288] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:08.189949 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.187453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42611 (local address 127.0.98.3:36545)
0925 14:49:08.187517 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.187519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.187533 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:49:08.187558 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.187559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.187568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.187642 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.187823 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.188313 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.189238 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.189597 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.189684 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.189694 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.189848 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.189885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:49:08.189970 2354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.187347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.2:42611)
0925 14:49:08.187411 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.187424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.187476 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.187661 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.187662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.187664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.187736 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.187738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.188402 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.188403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.189138 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.189140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.189692 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.189693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.189772 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.189775 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.189841 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.189879 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:49:08.190238 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.187391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42395 (local address 127.0.98.1:33999)
0925 14:49:08.187437 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.187439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.187724 (+ 285us) server_negotiation.cc:407] Connection header received
0925 14:49:08.187789 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.187790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.187801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.187870 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.188054 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.188515 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.189409 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.189814 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.189891 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.189901 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.190125 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.190175 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:49:08.190316 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b8d19188ea417b97261c283e43b94d" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c09721b37cb4f76b8ef919060930ae0" is_pre_election: true
I20260925 14:49:08.190482 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.187350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.98.2:42395)
0925 14:49:08.187641 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.187653 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.187729 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.187871 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.187872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.187873 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.187976 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.187978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.188601 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.188602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.189310 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.189312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.189891 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.189892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.190007 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.190031 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.190320 (+ 289us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.190370 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":112,"thread_start_us":133,"threads_started":1}
I20260925 14:49:08.190496 2555 raft_consensus.cc:2454] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 0.
I20260925 14:49:08.190798 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b8d19188ea417b97261c283e43b94d" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d" is_pre_election: true
I20260925 14:49:08.190824 2347 leader_election.cc:302] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7ac715ab954fae9c0db65f646386b0, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:08.190935 2275 raft_consensus.cc:2454] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 0.
I20260925 14:49:08.190939 2631 raft_consensus.cc:2781] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:08.191001 2631 raft_consensus.cc:482] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:08.191027 2631 raft_consensus.cc:3037] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.191917 2631 raft_consensus.cc:504] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.192067 2631 leader_election.cc:288] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 election: Requested vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:08.192215 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b8d19188ea417b97261c283e43b94d" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d"
I20260925 14:49:08.192225 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b8d19188ea417b97261c283e43b94d" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c09721b37cb4f76b8ef919060930ae0"
I20260925 14:49:08.192297 2275 raft_consensus.cc:3037] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.192292 2555 raft_consensus.cc:3037] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.193038 2555 raft_consensus.cc:2454] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 1.
I20260925 14:49:08.193055 2275 raft_consensus.cc:2454] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 1.
I20260925 14:49:08.193194 2347 leader_election.cc:302] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7ac715ab954fae9c0db65f646386b0, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:08.193274 2631 raft_consensus.cc:2781] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:08.193426 2631 raft_consensus.cc:686] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Running, Role: LEADER
I20260925 14:49:08.193540 2631 consensus_queue.cc:237] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.194350 2118 catalog_manager.cc:5495] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7d7ac715ab954fae9c0db65f646386b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: UNKNOWN } } }
W20260925 14:49:08.226028 2322 tablet.cc:2369] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:49:08 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:08.230906 2168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.223272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.0.98.62:42275)
0925 14:49:08.223387 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.223390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.223502 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:49:08.223602 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.223603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.223613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.223680 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:08.224233 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.224723 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.225418 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.225600 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.225684 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.227909 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:08.227911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:08.227911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:08.227918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:08.229592 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.230049 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.230050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.230050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.230101 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:08.230300 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:08.230301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:08.230301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:08.230486 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:08.230563 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.230721 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.230804 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 14:49:08.231025 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.223159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42275 (local address 127.0.0.1:46664)
0925 14:49:08.223412 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.223424 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.223523 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.223698 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.223699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.224076 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:08.224124 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.224127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.224805 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.224806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.225326 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.225327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.225684 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.225685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.225737 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.226412 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:08.226416 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:08.227801 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:08.229700 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:08.229707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:08.229710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:08.229956 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:08.230167 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:08.230168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:08.230168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:08.230223 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:08.230553 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:08.230560 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:08.230607 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.230700 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.230905 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":81,"thread_start_us":116,"threads_started":1}
I20260925 14:49:08.233156 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46664:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:49:08.233356 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:08.234907 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 06ae33660aaf4b868c1cc0d922fee1ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c78dc6473cc40f1ac99e0abcd983077]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:08.234987 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 06ae33660aaf4b868c1cc0d922fee1ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c78dc6473cc40f1ac99e0abcd983077]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:08.235010 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 06ae33660aaf4b868c1cc0d922fee1ac (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c78dc6473cc40f1ac99e0abcd983077]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:08.235091 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ae33660aaf4b868c1cc0d922fee1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.235154 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ae33660aaf4b868c1cc0d922fee1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.235155 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06ae33660aaf4b868c1cc0d922fee1ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.236347 2624 tablet_bootstrap.cc:492] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: Bootstrap starting.
I20260925 14:49:08.236595 2626 tablet_bootstrap.cc:492] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: Bootstrap starting.
I20260925 14:49:08.236595 2627 tablet_bootstrap.cc:492] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: Bootstrap starting.
I20260925 14:49:08.236739 2624 tablet_bootstrap.cc:654] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.237036 2627 tablet_bootstrap.cc:654] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.237143 2626 tablet_bootstrap.cc:654] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.237797 2627 tablet_bootstrap.cc:492] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: No bootstrap required, opened a new log
I20260925 14:49:08.237795 2626 tablet_bootstrap.cc:492] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: No bootstrap required, opened a new log
I20260925 14:49:08.237847 2627 ts_tablet_manager.cc:1397] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:08.237850 2626 ts_tablet_manager.cc:1397] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:08.237905 2624 tablet_bootstrap.cc:492] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: No bootstrap required, opened a new log
I20260925 14:49:08.237948 2624 ts_tablet_manager.cc:1397] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:08.237963 2627 raft_consensus.cc:348] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.238025 2627 raft_consensus.cc:374] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.238039 2627 raft_consensus.cc:729] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.238029 2626 raft_consensus.cc:348] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.238106 2626 raft_consensus.cc:374] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.238066 2624 raft_consensus.cc:348] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.238128 2626 raft_consensus.cc:729] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.238132 2624 raft_consensus.cc:374] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.238157 2624 raft_consensus.cc:729] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245397fad9f84c9db25cfdffd719083d, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.238142 2627 consensus_queue.cc:260] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.238181 2626 consensus_queue.cc:260] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.238204 2624 consensus_queue.cc:260] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.238242 2627 ts_tablet_manager.cc:1428] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:08.238271 2626 ts_tablet_manager.cc:1428] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:08.238302 2624 ts_tablet_manager.cc:1428] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:08.238277 2631 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.238339 2631 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:08.238407 2632 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.238495 2632 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:08.238615 2630 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.238714 2630 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 14:49:08.298107 2602 tablet.cc:2369] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:08.387764 2632 raft_consensus.cc:482] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:08.387880 2632 raft_consensus.cc:504] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.388319 2632 leader_election.cc:288] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491)
I20260925 14:49:08.392426 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.388983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33715 (local address 127.0.98.2:33491)
0925 14:49:08.389026 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.389028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.389036 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:08.389062 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.389063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.389074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.389135 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.389250 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.390075 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.391535 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.392060 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.392109 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.392119 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.392329 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.392369 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:08.392416 2494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.388498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.98.3:53299)
0925 14:49:08.388628 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.388640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.388687 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.388920 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.388921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.388922 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.389029 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.389031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.390138 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.390139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.391368 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.391371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.392149 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.392150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.392238 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.392243 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.392293 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.392341 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":1}
I20260925 14:49:08.392611 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.388498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33491 (local address 127.0.98.3:33715)
0925 14:49:08.388782 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.388789 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.388814 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.389139 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.389140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.389141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.389199 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.389200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.390173 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.390174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.391428 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.391430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.392158 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.392159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.392238 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.392242 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.392500 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.392544 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":47,"thread_start_us":98,"threads_started":1}
I20260925 14:49:08.392695 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.388695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53299 (local address 127.0.98.1:33999)
0925 14:49:08.388742 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.388745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.388754 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:08.388808 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.388809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.388820 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.388917 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.389139 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.390021 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.391472 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.392006 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.392115 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.392128 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.392567 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.392614 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:49:08.393070 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae33660aaf4b868c1cc0d922fee1ac" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7ac715ab954fae9c0db65f646386b0" is_pre_election: true
I20260925 14:49:08.393095 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae33660aaf4b868c1cc0d922fee1ac" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d" is_pre_election: true
I20260925 14:49:08.393230 2275 raft_consensus.cc:2454] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 0.
I20260925 14:49:08.393249 2415 raft_consensus.cc:2454] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 0.
I20260925 14:49:08.393447 2486 leader_election.cc:302] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7ac715ab954fae9c0db65f646386b0, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:08.393673 2632 raft_consensus.cc:2781] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:08.393723 2632 raft_consensus.cc:482] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:08.393749 2632 raft_consensus.cc:3037] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.394479 2632 raft_consensus.cc:504] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.394629 2632 leader_election.cc:288] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 election: Requested vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491)
I20260925 14:49:08.394757 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae33660aaf4b868c1cc0d922fee1ac" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d"
I20260925 14:49:08.394768 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06ae33660aaf4b868c1cc0d922fee1ac" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7ac715ab954fae9c0db65f646386b0"
I20260925 14:49:08.394830 2415 raft_consensus.cc:3037] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.394831 2275 raft_consensus.cc:3037] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.395550 2275 raft_consensus.cc:2454] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 1.
I20260925 14:49:08.395574 2415 raft_consensus.cc:2454] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 1.
I20260925 14:49:08.395722 2489 leader_election.cc:302] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245397fad9f84c9db25cfdffd719083d, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:08.395830 2632 raft_consensus.cc:2781] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:08.396031 2632 raft_consensus.cc:686] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Running, Role: LEADER
I20260925 14:49:08.396152 2632 consensus_queue.cc:237] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.396603 2646 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: TxnStatusTablet state changed. Reason: New leader 8c09721b37cb4f76b8ef919060930ae0. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.396687 2646 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:08.396855 2648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:49:08.396975 2648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:49:08.396998 2117 catalog_manager.cc:5495] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: HEALTHY } } }
I20260925 14:49:08.420383 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.417630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.98.3:36545)
0925 14:49:08.417671 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.417673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.417686 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:49:08.417774 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.417774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.417789 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.417845 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.417980 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.418522 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.419614 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.419786 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.419830 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.420034 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.420162 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.420218 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.420293 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.420312 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:08.420420 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.417552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.0.1:39310)
0925 14:49:08.417627 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.417640 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.417691 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.417867 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.417867 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.417869 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:08.417921 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.417923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.418594 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.418595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.419527 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.419530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.419853 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.419854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.419920 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.419936 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.420226 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.420233 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.420291 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.420338 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:49:08.422140 2415 raft_consensus.cc:1261] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Refusing update from remote peer 8c09721b37cb4f76b8ef919060930ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:08.422134 2275 raft_consensus.cc:1261] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Refusing update from remote peer 8c09721b37cb4f76b8ef919060930ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:08.422487 2646 consensus_queue.cc:1035] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:08.422624 2632 consensus_queue.cc:1035] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:08.423666 2631 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: TxnStatusTablet state changed. Reason: New leader 8c09721b37cb4f76b8ef919060930ae0. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.423758 2631 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:08.423802 2630 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: TxnStatusTablet state changed. Reason: New leader 8c09721b37cb4f76b8ef919060930ae0. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.423892 2630 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:08.424547 2632 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.424626 2632 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:08.424717 2631 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.424799 2631 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 7d7ac715ab954fae9c0db65f646386b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:08.425068 2653 mvcc.cc:205] Tried to move back new op lower bound from 7333264377534431232 to 7333264377431056384. Current Snapshot: MvccSnapshot[applied={T|T < 7333264377534431232}]
I20260925 14:49:08.425268 2646 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.425336 2646 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 8c09721b37cb4f76b8ef919060930ae0: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:08.425570 2630 tablet_replica.cc:439] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } } }
I20260925 14:49:08.425659 2630 tablet_replica.cc:442] T 06ae33660aaf4b868c1cc0d922fee1ac P 245397fad9f84c9db25cfdffd719083d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:08.430227 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.427689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.0.98.2:33491)
0925 14:49:08.427727 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.427729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.427751 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:49:08.427835 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.427836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.427852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.427914 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.428072 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.428624 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.429414 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.429658 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.429719 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.429909 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.430014 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.430108 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.430131 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.430152 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:08.435190 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.432696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34891 (local address 127.0.98.3:36545)
0925 14:49:08.432750 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.432752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.432762 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:08.432836 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.432836 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.432843 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.432889 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.433026 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.433465 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.434574 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.434745 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.434797 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.434993 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.435017 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.435051 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.435127 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.435142 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:49:08.435256 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.432552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.2:34891)
0925 14:49:08.432711 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.432723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.432774 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.432909 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.432910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.432911 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:08.432967 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.432969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.433530 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.433531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.434500 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.434502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.434816 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.434817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.434879 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.434895 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.435071 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.435073 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.435125 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.435167 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":80,"threads_started":1}
I20260925 14:49:08.437700 2275 raft_consensus.cc:1261] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Refusing update from remote peer 7d7ac715ab954fae9c0db65f646386b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:08.437763 2555 raft_consensus.cc:1261] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Refusing update from remote peer 7d7ac715ab954fae9c0db65f646386b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:08.438024 2634 consensus_queue.cc:1035] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:08.438158 2631 consensus_queue.cc:1035] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:08.440222 2651 mvcc.cc:205] Tried to move back new op lower bound from 7333264377599049728 to 7333264376601223168. Current Snapshot: MvccSnapshot[applied={T|T < 7333264377599049728}]
I20260925 14:49:08.446774 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.444130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33491 (local address 127.0.98.3:39783)
0925 14:49:08.444204 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.444216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.444264 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.444387 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.444387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.444388 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:08.444452 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.444453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.445062 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.445063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.445954 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.445957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.446334 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.446334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.446420 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.446434 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.446617 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.446619 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.446673 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.446712 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 14:49:08.446764 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.444183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39783 (local address 127.0.98.2:33491)
0925 14:49:08.444222 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.444224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.444248 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:49:08.444319 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.444320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.444331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.444383 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.444524 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.444978 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.446044 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.446247 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.446333 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.446518 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.446549 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.446617 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.446677 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.446695 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:08.456125 2117 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:08.456257 2117 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:08.456987 2117 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Sending DeleteTablet for 3 replicas of tablet 91b8d19188ea417b97261c283e43b94d
I20260925 14:49:08.457291 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b8d19188ea417b97261c283e43b94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39304
I20260925 14:49:08.457412 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b8d19188ea417b97261c283e43b94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52770
I20260925 14:49:08.457446 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b8d19188ea417b97261c283e43b94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52050
I20260925 14:49:08.457661 2671 tablet_replica.cc:330] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: stopping tablet replica
I20260925 14:49:08.457764 2671 raft_consensus.cc:2229] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:08.457902 2671 raft_consensus.cc:2258] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:08.457943 2117 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:08.458097 2117 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:08.458791 2671 ts_tablet_manager.cc:1905] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:08.459551 2255 tablet_service.cc:1467] Processing CreateTablet for tablet ac7ea90f54814160bd43f2cda752fc8e (DEFAULT_TABLE table=test-table [id=f656d9a1b80b4502a4cac7fab8cab09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:08.459684 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7ea90f54814160bd43f2cda752fc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.460986 2671 ts_tablet_manager.cc:1918] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:08.461082 2671 log.cc:1196] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/91b8d19188ea417b97261c283e43b94d
I20260925 14:49:08.461177 2624 tablet_bootstrap.cc:492] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Bootstrap starting.
I20260925 14:49:08.461398 2671 ts_tablet_manager.cc:1939] T 91b8d19188ea417b97261c283e43b94d P 7d7ac715ab954fae9c0db65f646386b0: Deleting consensus metadata
I20260925 14:49:08.461611 2624 tablet_bootstrap.cc:654] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.462066 2672 tablet_replica.cc:330] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: stopping tablet replica
I20260925 14:49:08.462121 2102 catalog_manager.cc:4841] TS 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491): tablet 91b8d19188ea417b97261c283e43b94d (table test-table [id=24f54c7707064f3ca7d14f796ead172f]) successfully deleted
I20260925 14:49:08.462226 2672 raft_consensus.cc:2229] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:08.462308 2672 raft_consensus.cc:2258] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:08.462473 2535 tablet_service.cc:1467] Processing CreateTablet for tablet ac7ea90f54814160bd43f2cda752fc8e (DEFAULT_TABLE table=test-table [id=f656d9a1b80b4502a4cac7fab8cab09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:08.462512 2670 tablet_replica.cc:330] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: stopping tablet replica
I20260925 14:49:08.462589 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7ea90f54814160bd43f2cda752fc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.462661 2670 raft_consensus.cc:2229] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:08.462734 2670 raft_consensus.cc:2258] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:08.462847 2672 ts_tablet_manager.cc:1905] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:08.463651 2626 tablet_bootstrap.cc:492] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Bootstrap starting.
I20260925 14:49:08.463905 2395 tablet_service.cc:1467] Processing CreateTablet for tablet ac7ea90f54814160bd43f2cda752fc8e (DEFAULT_TABLE table=test-table [id=f656d9a1b80b4502a4cac7fab8cab09a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:08.464035 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7ea90f54814160bd43f2cda752fc8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:08.464201 2626 tablet_bootstrap.cc:654] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.464370 2672 ts_tablet_manager.cc:1918] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:08.464429 2672 log.cc:1196] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/91b8d19188ea417b97261c283e43b94d
I20260925 14:49:08.464668 2672 ts_tablet_manager.cc:1939] T 91b8d19188ea417b97261c283e43b94d P 245397fad9f84c9db25cfdffd719083d: Deleting consensus metadata
I20260925 14:49:08.465286 2627 tablet_bootstrap.cc:492] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Bootstrap starting.
I20260925 14:49:08.465430 2105 catalog_manager.cc:4841] TS 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999): tablet 91b8d19188ea417b97261c283e43b94d (table test-table [id=24f54c7707064f3ca7d14f796ead172f]) successfully deleted
I20260925 14:49:08.465716 2627 tablet_bootstrap.cc:654] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:08.466558 2670 ts_tablet_manager.cc:1905] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:08.466810 2627 tablet_bootstrap.cc:492] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: No bootstrap required, opened a new log
I20260925 14:49:08.466866 2627 ts_tablet_manager.cc:1397] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:49:08.466997 2627 raft_consensus.cc:348] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.467057 2627 raft_consensus.cc:374] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.467077 2627 raft_consensus.cc:729] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.467124 2627 consensus_queue.cc:260] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.467211 2627 ts_tablet_manager.cc:1428] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:08.467475 2624 tablet_bootstrap.cc:492] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: No bootstrap required, opened a new log
I20260925 14:49:08.467527 2624 ts_tablet_manager.cc:1397] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260925 14:49:08.467661 2624 raft_consensus.cc:348] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.467731 2624 raft_consensus.cc:374] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.467751 2624 raft_consensus.cc:729] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245397fad9f84c9db25cfdffd719083d, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.467793 2624 consensus_queue.cc:260] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.467871 2624 ts_tablet_manager.cc:1428] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:08.469187 2670 ts_tablet_manager.cc:1918] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:08.469290 2670 log.cc:1196] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/91b8d19188ea417b97261c283e43b94d
I20260925 14:49:08.469640 2670 ts_tablet_manager.cc:1939] T 91b8d19188ea417b97261c283e43b94d P 8c09721b37cb4f76b8ef919060930ae0: Deleting consensus metadata
I20260925 14:49:08.470248 2103 catalog_manager.cc:4841] TS 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545): tablet 91b8d19188ea417b97261c283e43b94d (table test-table [id=24f54c7707064f3ca7d14f796ead172f]) successfully deleted
I20260925 14:49:08.470759 2626 tablet_bootstrap.cc:492] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: No bootstrap required, opened a new log
I20260925 14:49:08.470829 2626 ts_tablet_manager.cc:1397] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 14:49:08.470953 2626 raft_consensus.cc:348] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.471041 2626 raft_consensus.cc:374] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:08.471071 2626 raft_consensus.cc:729] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Initialized, Role: FOLLOWER
I20260925 14:49:08.471119 2626 consensus_queue.cc:260] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.471221 2626 ts_tablet_manager.cc:1428] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:08.766985 2630 raft_consensus.cc:482] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:08.767091 2630 raft_consensus.cc:504] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.767375 2630 leader_election.cc:288] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491), 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:08.771883 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.767660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37101 (local address 127.0.98.3:36545)
0925 14:49:08.767709 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.767712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.767792 (+ 80us) server_negotiation.cc:407] Connection header received
0925 14:49:08.767845 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.767846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.767855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.767909 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.768133 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.768864 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.770962 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.771519 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.771609 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.771619 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.771768 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.771806 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:08.771881 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.767663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34001 (local address 127.0.98.2:33491)
0925 14:49:08.767709 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.767712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.767792 (+ 80us) server_negotiation.cc:407] Connection header received
0925 14:49:08.767844 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.767845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.767855 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.767930 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:08.768135 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.768873 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.770962 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.771532 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.771602 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.771613 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.771768 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.771805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:08.771978 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.767579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.1:37101)
0925 14:49:08.767705 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.767717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.767786 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.767955 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.767956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.767958 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.768041 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.768043 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.769033 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.769034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.770858 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.770862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.771619 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.771619 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.771690 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.771692 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.771746 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.771806 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":72,"thread_start_us":58,"threads_started":1}
I20260925 14:49:08.771984 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.767579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33491 (local address 127.0.98.1:34001)
0925 14:49:08.767705 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.767717 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.767786 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.767956 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.767956 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.767959 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:08.768041 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.768044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.769004 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.769005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.770858 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.770862 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.771615 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.771616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.771686 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.771691 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.771746 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.771806 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":97,"thread_start_us":60,"threads_started":1}
I20260925 14:49:08.772281 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7ea90f54814160bd43f2cda752fc8e" candidate_uuid: "245397fad9f84c9db25cfdffd719083d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c09721b37cb4f76b8ef919060930ae0" is_pre_election: true
I20260925 14:49:08.772390 2555 raft_consensus.cc:2454] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 245397fad9f84c9db25cfdffd719083d in term 0.
I20260925 14:49:08.772471 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7ea90f54814160bd43f2cda752fc8e" candidate_uuid: "245397fad9f84c9db25cfdffd719083d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7ac715ab954fae9c0db65f646386b0" is_pre_election: true
I20260925 14:49:08.772572 2415 raft_consensus.cc:2454] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 245397fad9f84c9db25cfdffd719083d in term 0.
I20260925 14:49:08.772575 2207 leader_election.cc:302] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245397fad9f84c9db25cfdffd719083d, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:08.772744 2630 raft_consensus.cc:2781] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:08.772795 2630 raft_consensus.cc:482] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:08.772818 2630 raft_consensus.cc:3037] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.773499 2630 raft_consensus.cc:504] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.773665 2630 leader_election.cc:288] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [CANDIDATE]: Term 1 election: Requested vote from peers 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491), 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:08.773844 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7ea90f54814160bd43f2cda752fc8e" candidate_uuid: "245397fad9f84c9db25cfdffd719083d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7ac715ab954fae9c0db65f646386b0"
I20260925 14:49:08.773871 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7ea90f54814160bd43f2cda752fc8e" candidate_uuid: "245397fad9f84c9db25cfdffd719083d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c09721b37cb4f76b8ef919060930ae0"
I20260925 14:49:08.773928 2555 raft_consensus.cc:3037] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.773944 2415 raft_consensus.cc:3037] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:08.774581 2555 raft_consensus.cc:2454] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 245397fad9f84c9db25cfdffd719083d in term 1.
I20260925 14:49:08.774647 2415 raft_consensus.cc:2454] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 245397fad9f84c9db25cfdffd719083d in term 1.
I20260925 14:49:08.774745 2207 leader_election.cc:302] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245397fad9f84c9db25cfdffd719083d, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:08.774847 2630 raft_consensus.cc:2781] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:08.774971 2630 raft_consensus.cc:686] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 1 LEADER]: Becoming Leader. State: Replica: 245397fad9f84c9db25cfdffd719083d, State: Running, Role: LEADER
I20260925 14:49:08.775044 2630 consensus_queue.cc:237] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:08.775825 2117 catalog_manager.cc:5495] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d reported cstate change: term changed from 0 to 1, leader changed from <none> to 245397fad9f84c9db25cfdffd719083d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "245397fad9f84c9db25cfdffd719083d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:08.824499 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.821960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.0.98.1:33999)
0925 14:49:08.822029 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.822032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.822075 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:49:08.822160 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.822161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.822182 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.822274 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.822415 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.822920 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.823702 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.823906 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.823984 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.824150 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.824261 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.824346 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.824388 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.824407 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:49:08.829780 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.826871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49701 (local address 127.0.98.3:36545)
0925 14:49:08.826940 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.826943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.827058 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:49:08.827124 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.827125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.827135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.827190 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.827314 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.827774 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.829045 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.829268 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.829346 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.829526 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.829565 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.829627 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.829683 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.829703 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:49:08.829808 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.826780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.1:49701)
0925 14:49:08.826985 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.826996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.827073 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.827193 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.827194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.827196 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:08.827256 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.827257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.827856 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.827857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.828945 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.828948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.829365 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.829366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.829426 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.829439 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.829630 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.829632 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.829680 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.829724 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
I20260925 14:49:08.831867 2555 raft_consensus.cc:1261] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Refusing update from remote peer 245397fad9f84c9db25cfdffd719083d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:08.831980 2415 raft_consensus.cc:1261] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Refusing update from remote peer 245397fad9f84c9db25cfdffd719083d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:08.832150 2678 consensus_queue.cc:1035] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:08.832259 2630 consensus_queue.cc:1035] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:08.833905 2652 mvcc.cc:205] Tried to move back new op lower bound from 7333264379213717504 to 7333264378983206912. Current Snapshot: MvccSnapshot[applied={T|T < 7333264379213717504}]
I20260925 14:49:08.839428 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.837139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52769 (local address 127.0.98.1:33999)
0925 14:49:08.837176 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:08.837178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:08.837189 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:08.837255 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:08.837256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:08.837263 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:08.837309 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:08.837426 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.838030 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.838760 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.838934 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.838985 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.839165 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.839191 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.839252 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:08.839329 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:49:08.839352 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:49:08.839435 2607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:08.837086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.98.3:52769)
0925 14:49:08.837151 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:08.837163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:08.837201 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:08.837337 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:08.837338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:08.837339 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:08.837380 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.837381 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.838104 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.838105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.838693 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:08.838695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:08.839016 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:08.839018 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:08.839070 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:08.839074 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:08.839266 (+ 192us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:08.839268 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:08.839309 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 14:49:08.839357 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 14:49:08.839828 2652 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:49:10.358692 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:10.358790 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:10.359571 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Sending DeleteTablet for 3 replicas of tablet ac7ea90f54814160bd43f2cda752fc8e
I20260925 14:49:10.359793 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7ea90f54814160bd43f2cda752fc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52770
I20260925 14:49:10.359810 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7ea90f54814160bd43f2cda752fc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52050
I20260925 14:49:10.359848 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet ac7ea90f54814160bd43f2cda752fc8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39304
I20260925 14:49:10.359992 2711 tablet_replica.cc:330] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: stopping tablet replica
I20260925 14:49:10.360031 2712 tablet_replica.cc:330] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: stopping tablet replica
I20260925 14:49:10.360083 2711 raft_consensus.cc:2229] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:10.360083 2712 raft_consensus.cc:2229] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:10.360117 2711 raft_consensus.cc:2258] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:10.360117 2712 raft_consensus.cc:2258] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:10.360445 2712 ts_tablet_manager.cc:1905] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:10.360466 2710 tablet_replica.cc:330] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: stopping tablet replica
I20260925 14:49:10.360512 2710 raft_consensus.cc:2229] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:10.360520 2711 ts_tablet_manager.cc:1905] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:10.360591 2710 raft_consensus.cc:2258] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:10.360796 2710 ts_tablet_manager.cc:1905] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:10.362296 2712 ts_tablet_manager.cc:1918] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:49:10.362350 2712 log.cc:1196] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/ac7ea90f54814160bd43f2cda752fc8e
I20260925 14:49:10.362449 2711 ts_tablet_manager.cc:1918] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:49:10.362437 2710 ts_tablet_manager.cc:1918] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:49:10.362509 2711 log.cc:1196] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/ac7ea90f54814160bd43f2cda752fc8e
I20260925 14:49:10.362528 2710 log.cc:1196] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/ac7ea90f54814160bd43f2cda752fc8e
I20260925 14:49:10.362646 2712 ts_tablet_manager.cc:1939] T ac7ea90f54814160bd43f2cda752fc8e P 8c09721b37cb4f76b8ef919060930ae0: Deleting consensus metadata
I20260925 14:49:10.362746 2710 ts_tablet_manager.cc:1939] T ac7ea90f54814160bd43f2cda752fc8e P 245397fad9f84c9db25cfdffd719083d: Deleting consensus metadata
I20260925 14:49:10.362752 2711 ts_tablet_manager.cc:1939] T ac7ea90f54814160bd43f2cda752fc8e P 7d7ac715ab954fae9c0db65f646386b0: Deleting consensus metadata
I20260925 14:49:10.362993 2103 catalog_manager.cc:4841] TS 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545): tablet ac7ea90f54814160bd43f2cda752fc8e (table test-table [id=f656d9a1b80b4502a4cac7fab8cab09a]) successfully deleted
I20260925 14:49:10.363072 2105 catalog_manager.cc:4841] TS 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999): tablet ac7ea90f54814160bd43f2cda752fc8e (table test-table [id=f656d9a1b80b4502a4cac7fab8cab09a]) successfully deleted
I20260925 14:49:10.363077 2102 catalog_manager.cc:4841] TS 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491): tablet ac7ea90f54814160bd43f2cda752fc8e (table test-table [id=f656d9a1b80b4502a4cac7fab8cab09a]) successfully deleted
Sep 25 14:49:10 dist-test-slave-z2s5 krb5kdc[2073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:49:10 dist-test-slave-z2s5 krb5kdc[2073](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:10.385150 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:10.377633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.0.98.62:42275)
0925 14:49:10.377776 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:10.377777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:10.377909 (+ 132us) server_negotiation.cc:407] Connection header received
0925 14:49:10.377963 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:10.377963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:10.377974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:10.378051 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:10.378721 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.379238 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.379995 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.380196 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.380247 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:10.382635 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:10.382637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:10.382637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:10.382643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:10.384156 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:10.384528 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:10.384528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:10.384529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:10.384580 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:10.384745 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:10.384746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:10.384746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:10.384842 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:10.384912 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:10.385010 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 14:49:10.385081 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":90,"threads_started":1}
I20260925 14:49:10.387239 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:10.387428 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:10.388988 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 (DEFAULT_TABLE table=test-table [id=bd08ed3211df4ae9ab36a903230e889a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:10.388988 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 (DEFAULT_TABLE table=test-table [id=bd08ed3211df4ae9ab36a903230e889a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:10.388998 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 (DEFAULT_TABLE table=test-table [id=bd08ed3211df4ae9ab36a903230e889a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:10.389132 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:10.389132 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:10.389143 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:10.390561 2721 tablet_bootstrap.cc:492] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Bootstrap starting.
I20260925 14:49:10.390945 2723 tablet_bootstrap.cc:492] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Bootstrap starting.
I20260925 14:49:10.390949 2721 tablet_bootstrap.cc:654] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:10.391315 2723 tablet_bootstrap.cc:654] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:10.391615 2722 tablet_bootstrap.cc:492] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Bootstrap starting.
I20260925 14:49:10.391657 2721 tablet_bootstrap.cc:492] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: No bootstrap required, opened a new log
I20260925 14:49:10.391716 2721 ts_tablet_manager.cc:1397] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:10.391873 2723 tablet_bootstrap.cc:492] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: No bootstrap required, opened a new log
I20260925 14:49:10.391835 2721 raft_consensus.cc:348] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.391918 2721 raft_consensus.cc:374] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:10.391932 2723 ts_tablet_manager.cc:1397] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:10.391952 2721 raft_consensus.cc:729] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:10.391973 2722 tablet_bootstrap.cc:654] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:10.391999 2721 consensus_queue.cc:260] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.392059 2723 raft_consensus.cc:348] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.392141 2723 raft_consensus.cc:374] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:10.392171 2723 raft_consensus.cc:729] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Initialized, Role: FOLLOWER
I20260925 14:49:10.392191 2721 ts_tablet_manager.cc:1428] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:10.392256 2723 consensus_queue.cc:260] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.392359 2723 ts_tablet_manager.cc:1428] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:10.392645 2722 tablet_bootstrap.cc:492] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: No bootstrap required, opened a new log
I20260925 14:49:10.392741 2722 ts_tablet_manager.cc:1397] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:10.392905 2722 raft_consensus.cc:348] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.393054 2722 raft_consensus.cc:374] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:10.393138 2722 raft_consensus.cc:729] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245397fad9f84c9db25cfdffd719083d, State: Initialized, Role: FOLLOWER
I20260925 14:49:10.393239 2722 consensus_queue.cc:260] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.393424 2722 ts_tablet_manager.cc:1428] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:49:10.473824 2688 raft_consensus.cc:482] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:10.473920 2688 raft_consensus.cc:504] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.474123 2688 leader_election.cc:288] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491)
I20260925 14:49:10.474295 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc04ae14c5a40bc81f8c5847ff8f5e8" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d" is_pre_election: true
I20260925 14:49:10.474404 2275 raft_consensus.cc:2454] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 0.
I20260925 14:49:10.474331 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc04ae14c5a40bc81f8c5847ff8f5e8" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7ac715ab954fae9c0db65f646386b0" is_pre_election: true
I20260925 14:49:10.474498 2415 raft_consensus.cc:2454] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 0.
I20260925 14:49:10.474602 2489 leader_election.cc:302] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245397fad9f84c9db25cfdffd719083d, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:10.474733 2688 raft_consensus.cc:2781] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:10.474776 2688 raft_consensus.cc:482] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:10.474789 2688 raft_consensus.cc:3037] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:10.475492 2688 raft_consensus.cc:504] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.475630 2688 leader_election.cc:288] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 election: Requested vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491)
I20260925 14:49:10.475785 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc04ae14c5a40bc81f8c5847ff8f5e8" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d"
I20260925 14:49:10.475790 2415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dc04ae14c5a40bc81f8c5847ff8f5e8" candidate_uuid: "8c09721b37cb4f76b8ef919060930ae0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7ac715ab954fae9c0db65f646386b0"
I20260925 14:49:10.475870 2415 raft_consensus.cc:3037] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:10.475871 2275 raft_consensus.cc:3037] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:10.476593 2415 raft_consensus.cc:2454] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 1.
I20260925 14:49:10.476783 2275 raft_consensus.cc:2454] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c09721b37cb4f76b8ef919060930ae0 in term 1.
I20260925 14:49:10.476783 2486 leader_election.cc:302] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7ac715ab954fae9c0db65f646386b0, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:10.476900 2688 raft_consensus.cc:2781] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:10.476955 2688 raft_consensus.cc:686] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Running, Role: LEADER
I20260925 14:49:10.477016 2688 consensus_queue.cc:237] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.477650 2117 catalog_manager.cc:5495] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8c09721b37cb4f76b8ef919060930ae0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: HEALTHY } } }
I20260925 14:49:10.491514 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:10.489011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39320 (local address 127.0.98.3:36545)
0925 14:49:10.489164 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:10.489166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:10.489173 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:10.489197 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:10.489197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:10.489206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:10.489279 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:10.489384 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.489937 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.490784 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.491018 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.491104 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:10.491250 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:10.491287 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:10.491346 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:10.491405 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:49:10.491429 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260925 14:49:10.495749 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:10.493692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60445 (local address 127.0.98.3:36545)
0925 14:49:10.493725 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:10.493727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:10.493784 (+ 57us) server_negotiation.cc:407] Connection header received
0925 14:49:10.493834 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:10.493835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:10.493841 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:10.493906 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:10.494005 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.494499 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.495152 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.495319 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.495365 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:10.495501 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:10.495521 (+ 20us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:10.495584 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:10.495654 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:49:10.495672 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:49:10.495765 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:10.493578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36545 (local address 127.0.98.3:60445)
0925 14:49:10.493729 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:10.493739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:10.493796 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:10.493927 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:10.493927 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:10.493929 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:10.493961 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:10.493962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.494559 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.494560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:10.495088 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:10.495089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.495387 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.495388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:10.495440 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:10.495444 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:10.495595 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:10.495597 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:10.495651 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:49:10.495693 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":99,"threads_started":1}
I20260925 14:49:10.497566 2275 raft_consensus.cc:1261] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Refusing update from remote peer 8c09721b37cb4f76b8ef919060930ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:10.497557 2415 raft_consensus.cc:1261] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Refusing update from remote peer 8c09721b37cb4f76b8ef919060930ae0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:10.497767 2691 consensus_queue.cc:1035] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:10.497910 2696 consensus_queue.cc:1035] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:10.507858 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:10.507964 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:10.508548 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Sending DeleteTablet for 3 replicas of tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8
I20260925 14:49:10.508785 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52770
I20260925 14:49:10.508816 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39304
I20260925 14:49:10.508823 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:10 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52050
I20260925 14:49:10.508874 2710 tablet_replica.cc:330] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: stopping tablet replica
I20260925 14:49:10.508903 2711 tablet_replica.cc:330] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: stopping tablet replica
I20260925 14:49:10.508924 2710 raft_consensus.cc:2229] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:10.508955 2711 raft_consensus.cc:2229] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:10.508960 2710 raft_consensus.cc:2258] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:10.508966 2712 tablet_replica.cc:330] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: stopping tablet replica
I20260925 14:49:10.508989 2711 raft_consensus.cc:2258] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:10.509025 2712 raft_consensus.cc:2229] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:10.509115 2712 raft_consensus.cc:2258] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:10.509413 2712 ts_tablet_manager.cc:1905] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:10.509421 2710 ts_tablet_manager.cc:1905] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:10.509369 2116 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:10.509508 2116 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:10.510535 2711 ts_tablet_manager.cc:1905] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:10.510632 2535 tablet_service.cc:1467] Processing CreateTablet for tablet 0e18d6672e24486996b8caad1a3e4455 (DEFAULT_TABLE table=test-table [id=c5954fb575024d068a201c90f52f8a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:10.510740 2535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e18d6672e24486996b8caad1a3e4455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:10.511065 2395 tablet_service.cc:1467] Processing CreateTablet for tablet 0e18d6672e24486996b8caad1a3e4455 (DEFAULT_TABLE table=test-table [id=c5954fb575024d068a201c90f52f8a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:10.511166 2395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e18d6672e24486996b8caad1a3e4455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:10.511284 2712 ts_tablet_manager.cc:1918] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:10.511327 2712 log.cc:1196] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/9dc04ae14c5a40bc81f8c5847ff8f5e8
I20260925 14:49:10.511543 2712 ts_tablet_manager.cc:1939] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 8c09721b37cb4f76b8ef919060930ae0: Deleting consensus metadata
I20260925 14:49:10.511811 2103 catalog_manager.cc:4841] TS 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545): tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 (table test-table [id=bd08ed3211df4ae9ab36a903230e889a]) successfully deleted
I20260925 14:49:10.511967 2711 ts_tablet_manager.cc:1918] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:10.512017 2711 log.cc:1196] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/9dc04ae14c5a40bc81f8c5847ff8f5e8
I20260925 14:49:10.512243 2711 ts_tablet_manager.cc:1939] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 7d7ac715ab954fae9c0db65f646386b0: Deleting consensus metadata
I20260925 14:49:10.512409 2255 tablet_service.cc:1467] Processing CreateTablet for tablet 0e18d6672e24486996b8caad1a3e4455 (DEFAULT_TABLE table=test-table [id=c5954fb575024d068a201c90f52f8a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:10.512519 2255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e18d6672e24486996b8caad1a3e4455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:10.512630 2721 tablet_bootstrap.cc:492] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Bootstrap starting.
I20260925 14:49:10.512930 2721 tablet_bootstrap.cc:654] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:10.512985 2723 tablet_bootstrap.cc:492] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Bootstrap starting.
I20260925 14:49:10.513329 2723 tablet_bootstrap.cc:654] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:10.513710 2722 tablet_bootstrap.cc:492] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d: Bootstrap starting.
I20260925 14:49:10.513972 2722 tablet_bootstrap.cc:654] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:10.514207 2710 ts_tablet_manager.cc:1918] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:10.514266 2710 log.cc:1196] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/9dc04ae14c5a40bc81f8c5847ff8f5e8
I20260925 14:49:10.514503 2710 ts_tablet_manager.cc:1939] T 9dc04ae14c5a40bc81f8c5847ff8f5e8 P 245397fad9f84c9db25cfdffd719083d: Deleting consensus metadata
I20260925 14:49:10.514535 2722 tablet_bootstrap.cc:492] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d: No bootstrap required, opened a new log
I20260925 14:49:10.514571 2722 ts_tablet_manager.cc:1397] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:10.514647 2722 raft_consensus.cc:348] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.514725 2722 raft_consensus.cc:374] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:10.514751 2722 raft_consensus.cc:729] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 245397fad9f84c9db25cfdffd719083d, State: Initialized, Role: FOLLOWER
I20260925 14:49:10.514797 2722 consensus_queue.cc:260] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.514885 2722 ts_tablet_manager.cc:1428] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:10.514941 2105 catalog_manager.cc:4841] TS 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999): tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 (table test-table [id=bd08ed3211df4ae9ab36a903230e889a]) successfully deleted
I20260925 14:49:10.515036 2721 tablet_bootstrap.cc:492] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: No bootstrap required, opened a new log
I20260925 14:49:10.515093 2721 ts_tablet_manager.cc:1397] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:10.515096 2102 catalog_manager.cc:4841] TS 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491): tablet 9dc04ae14c5a40bc81f8c5847ff8f5e8 (table test-table [id=bd08ed3211df4ae9ab36a903230e889a]) successfully deleted
I20260925 14:49:10.515196 2721 raft_consensus.cc:348] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.515262 2721 raft_consensus.cc:374] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:10.515283 2721 raft_consensus.cc:729] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:10.515326 2721 consensus_queue.cc:260] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.515424 2721 ts_tablet_manager.cc:1428] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:10.520990 2723 tablet_bootstrap.cc:492] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: No bootstrap required, opened a new log
I20260925 14:49:10.521070 2723 ts_tablet_manager.cc:1397] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260925 14:49:10.521184 2723 raft_consensus.cc:348] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.521257 2723 raft_consensus.cc:374] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:10.521289 2723 raft_consensus.cc:729] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c09721b37cb4f76b8ef919060930ae0, State: Initialized, Role: FOLLOWER
I20260925 14:49:10.521346 2723 consensus_queue.cc:260] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.521436 2723 ts_tablet_manager.cc:1428] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:10.619268 2726 raft_consensus.cc:482] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:10.619372 2726 raft_consensus.cc:504] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.619635 2726 leader_election.cc:288] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:10.619843 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e18d6672e24486996b8caad1a3e4455" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d" is_pre_election: true
I20260925 14:49:10.619839 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e18d6672e24486996b8caad1a3e4455" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c09721b37cb4f76b8ef919060930ae0" is_pre_election: true
I20260925 14:49:10.619932 2275 raft_consensus.cc:2454] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 0.
I20260925 14:49:10.619956 2555 raft_consensus.cc:2454] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 0.
I20260925 14:49:10.620098 2349 leader_election.cc:302] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 245397fad9f84c9db25cfdffd719083d, 7d7ac715ab954fae9c0db65f646386b0; no voters:
I20260925 14:49:10.620224 2726 raft_consensus.cc:2781] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:10.620280 2726 raft_consensus.cc:482] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:10.620296 2726 raft_consensus.cc:3037] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:10.621049 2726 raft_consensus.cc:504] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.621196 2726 leader_election.cc:288] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 election: Requested vote from peers 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999), 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545)
I20260925 14:49:10.621430 2555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e18d6672e24486996b8caad1a3e4455" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c09721b37cb4f76b8ef919060930ae0"
I20260925 14:49:10.621387 2275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e18d6672e24486996b8caad1a3e4455" candidate_uuid: "7d7ac715ab954fae9c0db65f646386b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "245397fad9f84c9db25cfdffd719083d"
I20260925 14:49:10.621521 2555 raft_consensus.cc:3037] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:10.621538 2275 raft_consensus.cc:3037] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:10.622187 2555 raft_consensus.cc:2454] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 1.
I20260925 14:49:10.622193 2275 raft_consensus.cc:2454] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d7ac715ab954fae9c0db65f646386b0 in term 1.
I20260925 14:49:10.622452 2347 leader_election.cc:302] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7ac715ab954fae9c0db65f646386b0, 8c09721b37cb4f76b8ef919060930ae0; no voters:
I20260925 14:49:10.622566 2726 raft_consensus.cc:2781] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:10.622696 2726 raft_consensus.cc:686] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 LEADER]: Becoming Leader. State: Replica: 7d7ac715ab954fae9c0db65f646386b0, State: Running, Role: LEADER
I20260925 14:49:10.622747 2726 consensus_queue.cc:237] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } }
I20260925 14:49:10.623417 2116 catalog_manager.cc:5495] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7d7ac715ab954fae9c0db65f646386b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d7ac715ab954fae9c0db65f646386b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:10.634716 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:10.632458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52072 (local address 127.0.98.2:33491)
0925 14:49:10.632608 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:10.632610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:10.632617 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:10.632635 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:10.632636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:10.632646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:10.632703 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:10.632816 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.633285 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.634066 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:10.634274 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:10.634361 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:10.634533 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:10.634561 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:10.634602 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:10.634637 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:49:10.634654 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260925 14:49:10.636835 2275 raft_consensus.cc:1261] T 0e18d6672e24486996b8caad1a3e4455 P 245397fad9f84c9db25cfdffd719083d [term 1 FOLLOWER]: Refusing update from remote peer 7d7ac715ab954fae9c0db65f646386b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:10.636847 2555 raft_consensus.cc:1261] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Refusing update from remote peer 7d7ac715ab954fae9c0db65f646386b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:10.637089 2726 consensus_queue.cc:1035] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "245397fad9f84c9db25cfdffd719083d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:10.637202 2740 consensus_queue.cc:1035] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c09721b37cb4f76b8ef919060930ae0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:49:10.641270 2692 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:49:12.160714 2116 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:12.160812 2116 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:12.161691 2116 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a08fdd0d4dd4d038745ac9250a8cced: Sending DeleteTablet for 3 replicas of tablet 0e18d6672e24486996b8caad1a3e4455
I20260925 14:49:12.161940 2255 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e18d6672e24486996b8caad1a3e4455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52770
I20260925 14:49:12.161966 2535 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e18d6672e24486996b8caad1a3e4455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39304
I20260925 14:49:12.161983 2395 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e18d6672e24486996b8caad1a3e4455 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52050
I20260925 14:49:12.162096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 2190
I20260925 14:49:12.163265 2768 tablet_replica.cc:330] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: stopping tablet replica
I20260925 14:49:12.163368 2768 raft_consensus.cc:2229] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:12.163486 2768 raft_consensus.cc:2258] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:12.163530 2769 tablet_replica.cc:330] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: stopping tablet replica
I20260925 14:49:12.163590 2769 raft_consensus.cc:2229] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:12.163668 2769 raft_consensus.cc:2258] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:12.163937 2768 ts_tablet_manager.cc:1905] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:12.165956 2768 ts_tablet_manager.cc:1918] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:49:12.166025 2768 log.cc:1196] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/0e18d6672e24486996b8caad1a3e4455
I20260925 14:49:12.166241 2768 ts_tablet_manager.cc:1939] T 0e18d6672e24486996b8caad1a3e4455 P 8c09721b37cb4f76b8ef919060930ae0: Deleting consensus metadata
I20260925 14:49:12.166575 2769 ts_tablet_manager.cc:1905] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:12.166716 2103 catalog_manager.cc:4841] TS 8c09721b37cb4f76b8ef919060930ae0 (127.0.98.3:36545): tablet 0e18d6672e24486996b8caad1a3e4455 (table test-table [id=c5954fb575024d068a201c90f52f8a86]) successfully deleted
I20260925 14:49:12.168154 2769 ts_tablet_manager.cc:1918] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:49:12.168236 2769 log.cc:1196] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/0e18d6672e24486996b8caad1a3e4455
I20260925 14:49:12.168563 2769 ts_tablet_manager.cc:1939] T 0e18d6672e24486996b8caad1a3e4455 P 7d7ac715ab954fae9c0db65f646386b0: Deleting consensus metadata
W20260925 14:49:12.168720 2105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33999 (error 108)
I20260925 14:49:12.168877 2102 catalog_manager.cc:4841] TS 7d7ac715ab954fae9c0db65f646386b0 (127.0.98.2:33491): tablet 0e18d6672e24486996b8caad1a3e4455 (table test-table [id=c5954fb575024d068a201c90f52f8a86]) successfully deleted
I20260925 14:49:12.169269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 2330
I20260925 14:49:12.169317 2772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:12.169101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33999 (local address 127.0.0.1:52796)
0925 14:49:12.169233 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:12.169280 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33999: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
W20260925 14:49:12.171265 2105 catalog_manager.cc:4584] TS 245397fad9f84c9db25cfdffd719083d (127.0.98.1:33999): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0e18d6672e24486996b8caad1a3e4455: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33999: connect: Connection refused (error 111)
I20260925 14:49:12.175446 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 2470
I20260925 14:49:12.182092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 2086
2026-09-25T14:49:12Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5979 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:12 dist-test-slave-z2s5 krb5kdc[2775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:12 dist-test-slave-z2s5 krb5kdc[2775](info): set up 2 sockets
Sep 25 14:49:12 dist-test-slave-z2s5 krb5kdc[2775](info): commencing operation
krb5kdc: starting...
W20260925 14:49:13.218590 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:13.337921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:13.417122 2788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:13.417316 2788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:13.417337 2788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:13.418526 2788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:13.418572 2788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:13.418587 2788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:13.418601 2788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:13.418612 2788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:13.420223 2788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:13.420475 2788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:13.420715 2788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:13.425661 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.425673 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.425673 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:13.425854 2788 server_base.cc:1034] running on GCE node
I20260925 14:49:13.426141 2788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:13.426501 2788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:13.427650 2788 hybrid_clock.cc:648] HybridClock initialized: now 1790347753427638 us; error 31 us; skew 500 ppm
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:13.437131 2788 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:13.437644 2788 webserver.cc:460] Webserver started at http://127.0.98.62:46837/ using document root <none> and password file <none>
I20260925 14:49:13.437891 2788 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:13.437943 2788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:13.438119 2788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:13.439234 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "0f94cdc3ecba4ca6b752abc7348608f1"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "d61d6a8481121014bcb40bae4e8001de"
server_key_iv: "3a8726f3b8c8de301854e6d3f9d137d3"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.439603 2788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "0f94cdc3ecba4ca6b752abc7348608f1"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "d61d6a8481121014bcb40bae4e8001de"
server_key_iv: "3a8726f3b8c8de301854e6d3f9d137d3"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.441051 2788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:13.441792 2803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.441948 2788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:13.442042 2788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "0f94cdc3ecba4ca6b752abc7348608f1"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "d61d6a8481121014bcb40bae4e8001de"
server_key_iv: "3a8726f3b8c8de301854e6d3f9d137d3"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.442096 2788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:13.465804 2788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:13.468286 2788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:13.468402 2788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:13.472493 2788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39157
I20260925 14:49:13.472558 2855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39157 every 8 connection(s)
I20260925 14:49:13.473061 2788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:13.473318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 2788
I20260925 14:49:13.473428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:13.473619 392 external_mini_cluster.cc:1428] Setting key fc3740aeab383a3e969e218464aa2bf4
I20260925 14:49:13.473873 2856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:13.477137 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1: Bootstrap starting.
I20260925 14:49:13.477763 2856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:13.478041 2856 log.cc:824] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:13.478858 2856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1: No bootstrap required, opened a new log
I20260925 14:49:13.480580 2856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39157 } }
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:13.480738 2856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:13.480809 2856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f94cdc3ecba4ca6b752abc7348608f1, State: Initialized, Role: FOLLOWER
I20260925 14:49:13.480969 2856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39157 } }
I20260925 14:49:13.481045 2856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:13.481066 2856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:13.481133 2856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:13.481724 2856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39157 } }
I20260925 14:49:13.481830 2856 leader_election.cc:302] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f94cdc3ecba4ca6b752abc7348608f1; no voters:
I20260925 14:49:13.482015 2856 leader_election.cc:288] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:13.482048 2861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:13.482167 2861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [term 1 LEADER]: Becoming Leader. State: Replica: 0f94cdc3ecba4ca6b752abc7348608f1, State: Running, Role: LEADER
I20260925 14:49:13.482296 2856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:13.482542 2861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39157 } }
I20260925 14:49:13.483002 2863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f94cdc3ecba4ca6b752abc7348608f1. Latest consensus state: current_term: 1 leader_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39157 } } }
I20260925 14:49:13.482982 2862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f94cdc3ecba4ca6b752abc7348608f1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39157 } } }
I20260925 14:49:13.483119 2863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:13.483136 2862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1 [sys.catalog]: This master's current role is: LEADER
W20260925 14:49:13.484287 2877 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:49:13.484356 2877 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:49:13.484419 2870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:13.484581 2870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:13.485239 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:13.474121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60770 (local address 127.0.98.62:39157)
0925 14:49:13.474312 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:13.474322 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:13.474355 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:49:13.475033 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:13.475041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:13.475119 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:13.475280 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:13.475955 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.477324 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.478121 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.478745 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.478858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:13.481276 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:13.481280 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:13.481280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:13.481299 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:13.483349 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:13.483759 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:13.483763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:13.483763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:13.483871 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:13.484152 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:13.484153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:13.484153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:13.484412 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:13.484568 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:13.484904 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 14:49:13.485048 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260925 14:49:13.486779 2870 catalog_manager.cc:1347] Generated new cluster ID: 59692e5c45214fa6a0ae04cb13486f98
I20260925 14:49:13.486835 2870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:13.514534 2870 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:13.515291 2870 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:13.518682 2870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f94cdc3ecba4ca6b752abc7348608f1: Generated new TSK 0
I20260925 14:49:13.518874 2870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:13.549626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39157
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:13.629374 2882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:13.629617 2882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:13.629657 2882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:13.630834 2882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:13.630900 2882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:13.630947 2882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:13.632452 2882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:13.632718 2882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:13.632975 2882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:13.638055 2887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.637979 2888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:13.638195 2882 server_base.cc:1034] running on GCE node
W20260925 14:49:13.637982 2890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:13.638540 2882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:13.638742 2882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:13.639878 2882 hybrid_clock.cc:648] HybridClock initialized: now 1790347753639853 us; error 46 us; skew 500 ppm
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:13.642544 2882 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:13.642926 2882 webserver.cc:460] Webserver started at http://127.0.98.1:37155/ using document root <none> and password file <none>
I20260925 14:49:13.643137 2882 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:13.643180 2882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:13.643266 2882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:13.644322 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "8bb3546bc8304e4eba5b8a04eaec6560"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "47918dfcb415b5b9a70979309880268f"
server_key_iv: "bfff77deb221d79518db8894b910485a"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.644733 2882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8bb3546bc8304e4eba5b8a04eaec6560"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "47918dfcb415b5b9a70979309880268f"
server_key_iv: "bfff77deb221d79518db8894b910485a"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.646138 2882 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:13.646895 2897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.647061 2882 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.647149 2882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "8bb3546bc8304e4eba5b8a04eaec6560"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "47918dfcb415b5b9a70979309880268f"
server_key_iv: "bfff77deb221d79518db8894b910485a"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.647220 2882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:13.667985 2882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:13.670603 2882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:13.670765 2882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:13.671021 2882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:13.671350 2882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:13.671386 2882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.671409 2882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:13.671422 2882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.677260 2882 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45151
I20260925 14:49:13.677348 3010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45151 every 8 connection(s)
I20260925 14:49:13.677680 2882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:13.685053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 2882
I20260925 14:49:13.685150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:13.685319 392 external_mini_cluster.cc:1428] Setting key 6dbba7d69e3f9f938d23531ab2aa0ca5
I20260925 14:49:13.688166 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:13.678505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35587 (local address 127.0.98.62:39157)
0925 14:49:13.678579 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:13.678582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:13.678884 (+ 302us) server_negotiation.cc:407] Connection header received
0925 14:49:13.679806 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:13.679807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:13.679817 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:13.679880 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:13.680491 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.681009 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.682077 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.682296 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.682374 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:13.684737 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:13.684738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:13.684739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:13.684747 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:13.686370 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:13.686922 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:13.686923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:13.686924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:13.687016 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:13.687352 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:13.687353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:13.687353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:13.687499 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:13.687563 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:13.687927 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 14:49:13.688029 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:49:13.688297 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:13.678555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39157 (local address 127.0.98.1:35587)
0925 14:49:13.678727 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:13.678765 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:13.679698 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:13.679944 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:13.679948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:13.680207 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:13.680395 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:13.680400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.681118 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.681119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:13.681936 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:13.681938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.682413 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.682415 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:13.682574 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:13.683106 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:13.683114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:13.684632 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:13.686490 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:13.686497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:13.686499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:13.686818 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:13.687105 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:13.687106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:13.687107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:13.687262 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:13.687591 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:13.687598 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:13.687759 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 14:49:13.687878 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 14:49:13.688048 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260925 14:49:13.690541 3011 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39157
I20260925 14:49:13.690629 3011 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:13.690836 3011 heartbeater.cc:511] Master 127.0.98.62:39157 requested a full tablet report, sending...
I20260925 14:49:13.691430 2820 ts_manager.cc:194] Registered new tserver with Master: 8bb3546bc8304e4eba5b8a04eaec6560 (127.0.98.1:45151)
I20260925 14:49:13.692418 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35587
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:13.711220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39157
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:13.791471 3016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:13.791669 3016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:13.791692 3016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:13.792829 3016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:13.792872 3016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:13.792898 3016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:13.794510 3016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:13.794716 3016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:13.794912 3016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:13.799957 3016 server_base.cc:1034] running on GCE node
W20260925 14:49:13.799873 3022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.799876 3021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.799873 3024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:13.800331 3016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:13.800530 3016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:13.801710 3016 hybrid_clock.cc:648] HybridClock initialized: now 1790347753801699 us; error 28 us; skew 500 ppm
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:13.804618 3016 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:13.804986 3016 webserver.cc:460] Webserver started at http://127.0.98.2:38265/ using document root <none> and password file <none>
I20260925 14:49:13.805227 3016 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:13.805274 3016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:13.805421 3016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:13.806542 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "17693505eb434faeb29f3af4968f77fb"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "1438f2dd7664a5621436322f33cd6f28"
server_key_iv: "570e97823e0b2e6e08638d22132c34e1"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.806910 3016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "17693505eb434faeb29f3af4968f77fb"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "1438f2dd7664a5621436322f33cd6f28"
server_key_iv: "570e97823e0b2e6e08638d22132c34e1"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.808233 3016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:13.809010 3031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.809221 3016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:13.809334 3016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "17693505eb434faeb29f3af4968f77fb"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "1438f2dd7664a5621436322f33cd6f28"
server_key_iv: "570e97823e0b2e6e08638d22132c34e1"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.809412 3016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:13.831635 3016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:13.833774 3016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:13.833953 3016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:13.834201 3016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:13.834573 3016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:13.834635 3016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.834679 3016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:13.834707 3016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.840330 3016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37217
I20260925 14:49:13.840410 3144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37217 every 8 connection(s)
I20260925 14:49:13.840734 3016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:13.846277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3016
I20260925 14:49:13.846403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:13.846629 392 external_mini_cluster.cc:1428] Setting key 3e12d8f75c4e8f483e1c180519e74502
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:13.852000 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:13.841587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38225 (local address 127.0.98.62:39157)
0925 14:49:13.841635 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:13.841637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:13.841921 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:49:13.842904 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:13.842905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:13.842917 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:13.842982 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:13.843639 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.844317 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.845355 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.845561 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.845620 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:13.848368 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:13.848370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:13.848370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:13.848377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:13.850253 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:13.850795 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:13.850796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:13.850796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:13.850879 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:13.851262 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:13.851263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:13.851263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:13.851426 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:13.851514 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:13.851796 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 14:49:13.851878 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:13.852123 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:13.841570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39157 (local address 127.0.98.2:38225)
0925 14:49:13.841774 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:13.841831 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:13.842794 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:13.843038 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:13.843042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:13.843365 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:13.843552 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:13.843556 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.844397 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.844397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:13.845269 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:13.845271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:13.845656 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:13.845657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:13.845763 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:13.846357 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:13.846370 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:13.848255 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:13.850347 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:13.850354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:13.850357 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:13.850721 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:13.851044 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:13.851045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:13.851045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:13.851165 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:13.851508 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:13.851512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:13.851636 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 14:49:13.851722 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:49:13.851851 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260925 14:49:13.852664 3145 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39157
I20260925 14:49:13.852762 3145 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:13.852968 3145 heartbeater.cc:511] Master 127.0.98.62:39157 requested a full tablet report, sending...
I20260925 14:49:13.853379 2820 ts_manager.cc:194] Registered new tserver with Master: 17693505eb434faeb29f3af4968f77fb (127.0.98.2:37217)
I20260925 14:49:13.853780 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38225
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:13.873155 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39157
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:13.955727 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:13.955933 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:13.955966 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:13.957073 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:13.957110 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:13.957136 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:13.958714 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:13.958897 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:13.959079 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:13.963868 3158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.963852 3155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:13.964102 3156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:13.964375 3150 server_base.cc:1034] running on GCE node
I20260925 14:49:13.964593 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:13.964854 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:13.966048 3150 hybrid_clock.cc:648] HybridClock initialized: now 1790347753966030 us; error 25 us; skew 500 ppm
Sep 25 14:49:13 dist-test-slave-z2s5 krb5kdc[2775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:13.968583 3150 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:13.969086 3150 webserver.cc:460] Webserver started at http://127.0.98.3:37199/ using document root <none> and password file <none>
I20260925 14:49:13.969326 3150 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:13.969393 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:13.969542 3150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:13.970633 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "256c91f3a7644a8ca229b51029131c00"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "78f042542a178a574b5c0e03614354c7"
server_key_iv: "8279e4070d85f491d9bb1c964a9906d7"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.970989 3150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "256c91f3a7644a8ca229b51029131c00"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "78f042542a178a574b5c0e03614354c7"
server_key_iv: "8279e4070d85f491d9bb1c964a9906d7"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.972272 3150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:13.973084 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:13.973285 3150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:13.973382 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "256c91f3a7644a8ca229b51029131c00"
format_stamp: "Formatted at 2026-09-25 14:49:13 on dist-test-slave-z2s5"
server_key: "78f042542a178a574b5c0e03614354c7"
server_key_iv: "8279e4070d85f491d9bb1c964a9906d7"
server_key_version: "encryptionkey@0"
I20260925 14:49:13.973459 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:14.012543 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:14.014648 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:14.014810 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:14.015058 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:14.015367 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:14.015421 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:14.015463 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:14.015493 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:14.021384 3150 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39509
I20260925 14:49:14.021432 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39509 every 8 connection(s)
I20260925 14:49:14.021775 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:49:14 dist-test-slave-z2s5 krb5kdc[2775](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:14.028373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3150
I20260925 14:49:14.028483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:14.028697 392 external_mini_cluster.cc:1428] Setting key 52da687e003da07d617624294b697eed
I20260925 14:49:14.031852 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:14.022525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41503 (local address 127.0.98.62:39157)
0925 14:49:14.022575 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:14.022577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:14.022853 (+ 276us) server_negotiation.cc:407] Connection header received
0925 14:49:14.023820 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:14.023821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:14.023831 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:14.023884 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:14.024450 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:14.024941 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:14.026143 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:14.026333 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:14.026462 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:14.028839 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:14.028841 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:14.028841 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:14.028849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:14.030473 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:14.030888 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:14.030889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:14.030890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:14.030952 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:14.031212 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:14.031213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:14.031214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:14.031328 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:14.031404 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:14.031684 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:49:14.031757 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:14.032047 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:14.022577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39157 (local address 127.0.98.3:41503)
0925 14:49:14.022727 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:14.022763 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:14.023721 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:14.023939 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:14.023942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:14.024191 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:14.024376 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:14.024380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:14.025120 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:14.025122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:14.026036 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:14.026039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:14.026576 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:14.026577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:14.026682 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:14.027221 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:14.027229 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:14.028706 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:14.030568 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:14.030575 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:14.030577 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:14.030798 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:14.031028 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:14.031028 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:14.031029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:14.031136 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:14.031395 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:14.031400 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:14.031550 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 14:49:14.031664 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:49:14.031821 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260925 14:49:14.032517 3279 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39157
I20260925 14:49:14.032610 3279 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:14.032747 3279 heartbeater.cc:511] Master 127.0.98.62:39157 requested a full tablet report, sending...
I20260925 14:49:14.033079 2820 ts_manager.cc:194] Registered new tserver with Master: 256c91f3a7644a8ca229b51029131c00 (127.0.98.3:39509)
I20260925 14:49:14.033401 2820 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41503
I20260925 14:49:14.040429 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:49:14.051553 2859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:14.050616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60782 (local address 127.0.98.62:39157)
0925 14:49:14.050656 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:14.050658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:14.050800 (+ 142us) server_negotiation.cc:407] Connection header received
0925 14:49:14.050842 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:14.050843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:14.050851 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:14.050909 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:14.051445 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60782: BlockingRecv error: recv got EOF from 127.0.0.1:60782 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:49:14.052506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 2882
I20260925 14:49:14.056537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3016
I20260925 14:49:14.061040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3150
I20260925 14:49:14.065129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 2788
2026-09-25T14:49:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1881 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:14 dist-test-slave-z2s5 krb5kdc[3291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:14 dist-test-slave-z2s5 krb5kdc[3291](info): set up 2 sockets
Sep 25 14:49:14 dist-test-slave-z2s5 krb5kdc[3291](info): commencing operation
krb5kdc: starting...
W20260925 14:49:15.094662 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:15 dist-test-slave-z2s5 krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:15.218161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:15.299628 3304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:15.299827 3304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:15.299849 3304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:15.301127 3304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:15.301180 3304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:15.301196 3304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:15.301209 3304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:15.301221 3304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:15.303354 3304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:15.303622 3304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:15.303862 3304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:15.308732 3309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:15.308712 3312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:15.308732 3310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:15.309008 3304 server_base.cc:1034] running on GCE node
I20260925 14:49:15.309319 3304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:15.309592 3304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:15.310727 3304 hybrid_clock.cc:648] HybridClock initialized: now 1790347755310694 us; error 48 us; skew 500 ppm
Sep 25 14:49:15 dist-test-slave-z2s5 krb5kdc[3291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:15.313338 3304 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:15.313832 3304 webserver.cc:460] Webserver started at http://127.0.98.62:33809/ using document root <none> and password file <none>
I20260925 14:49:15.314109 3304 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:15.314157 3304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:15.314297 3304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:15.315416 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "677a23e13f2f4393b20839bd2b08e131"
format_stamp: "Formatted at 2026-09-25 14:49:15 on dist-test-slave-z2s5"
server_key: "1d7ab2ed131beb8992216673007c3403"
server_key_iv: "8ac436d007b533589816b7e61118e9ed"
server_key_version: "encryptionkey@0"
I20260925 14:49:15.315764 3304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "677a23e13f2f4393b20839bd2b08e131"
format_stamp: "Formatted at 2026-09-25 14:49:15 on dist-test-slave-z2s5"
server_key: "1d7ab2ed131beb8992216673007c3403"
server_key_iv: "8ac436d007b533589816b7e61118e9ed"
server_key_version: "encryptionkey@0"
I20260925 14:49:15.317162 3304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:15.317955 3319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:15.318280 3304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:15.318353 3304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "677a23e13f2f4393b20839bd2b08e131"
format_stamp: "Formatted at 2026-09-25 14:49:15 on dist-test-slave-z2s5"
server_key: "1d7ab2ed131beb8992216673007c3403"
server_key_iv: "8ac436d007b533589816b7e61118e9ed"
server_key_version: "encryptionkey@0"
I20260925 14:49:15.318463 3304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:15.339260 3304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:15.341676 3304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:15.341830 3304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:15.346202 3304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46069
I20260925 14:49:15.346289 3371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46069 every 8 connection(s)
I20260925 14:49:15.346612 3304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:15.347519 3372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:15.349742 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131: Bootstrap starting.
I20260925 14:49:15.350317 3372 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:15.350620 3372 log.cc:824] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:15.351306 3372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131: No bootstrap required, opened a new log
I20260925 14:49:15.352875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3304
I20260925 14:49:15.352960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:15.352869 3372 raft_consensus.cc:348] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a23e13f2f4393b20839bd2b08e131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260925 14:49:15.353070 3372 raft_consensus.cc:374] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:15.353134 3372 raft_consensus.cc:729] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 677a23e13f2f4393b20839bd2b08e131, State: Initialized, Role: FOLLOWER
I20260925 14:49:15.353189 392 external_mini_cluster.cc:1428] Setting key 375098c73931c1a3b80b4c592a561e29
I20260925 14:49:15.353289 3372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a23e13f2f4393b20839bd2b08e131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260925 14:49:15.353384 3372 raft_consensus.cc:388] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:15.353421 3372 raft_consensus.cc:482] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:15.353473 3372 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:15.354022 3372 raft_consensus.cc:504] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a23e13f2f4393b20839bd2b08e131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260925 14:49:15.354146 3372 leader_election.cc:302] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 677a23e13f2f4393b20839bd2b08e131; no voters:
I20260925 14:49:15.354329 3372 leader_election.cc:288] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:15.354388 3377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:15.354590 3377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [term 1 LEADER]: Becoming Leader. State: Replica: 677a23e13f2f4393b20839bd2b08e131, State: Running, Role: LEADER
I20260925 14:49:15.354602 3372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:15.354775 3377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a23e13f2f4393b20839bd2b08e131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } }
I20260925 14:49:15.355921 3378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "677a23e13f2f4393b20839bd2b08e131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a23e13f2f4393b20839bd2b08e131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } } }
I20260925 14:49:15.355928 3379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 677a23e13f2f4393b20839bd2b08e131. Latest consensus state: current_term: 1 leader_uuid: "677a23e13f2f4393b20839bd2b08e131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "677a23e13f2f4393b20839bd2b08e131" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46069 } } }
I20260925 14:49:15.356053 3378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:15.356058 3379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:15.356382 3392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:15.356878 3392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:15.358515 3392 catalog_manager.cc:1347] Generated new cluster ID: f4f879d49f2544feaf686951831631c4
I20260925 14:49:15.358577 3392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:15 dist-test-slave-z2s5 krb5kdc[3291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:15.366374 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:15.353677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52120 (local address 127.0.98.62:46069)
0925 14:49:15.354319 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:15.354328 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:15.354362 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:49:15.354504 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:15.354511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:15.354576 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:15.355613 (+ 1037us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:15.355709 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:15.357128 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:15.358038 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:15.358677 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:15.358783 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:15.361949 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:15.361953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:15.361954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:15.361967 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:15.364610 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:15.365096 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:15.365098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:15.365099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:15.365221 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:15.365467 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:15.365468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:15.365468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:15.365759 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:15.365895 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:15.366037 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 14:49:15.366166 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":513,"threads_started":1}
I20260925 14:49:15.367408 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:49:15.384100 3376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:15.379661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52132 (local address 127.0.98.62:46069)
0925 14:49:15.379707 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:15.379709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:15.379828 (+ 119us) server_negotiation.cc:407] Connection header received
0925 14:49:15.379890 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:15.379890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:15.379903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:15.379984 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:15.380488 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:15.381275 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:15.382293 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:15.382670 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:15.382750 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:15.383548 (+ 798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:15.383550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:15.383550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:15.383824 (+ 274us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:15.383844 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:15.383851 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:15.383914 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:15.383983 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:49:15.384011 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:15.392658 3392 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:15.393404 3392 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:15.397171 3392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 677a23e13f2f4393b20839bd2b08e131: Generated new TSK 0
I20260925 14:49:15.397420 3392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:15.400987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3304
2026-09-25T14:49:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1334 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:15 dist-test-slave-z2s5 krb5kdc[3405](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:15 dist-test-slave-z2s5 krb5kdc[3405](info): set up 2 sockets
Sep 25 14:49:15 dist-test-slave-z2s5 krb5kdc[3405](info): commencing operation
krb5kdc: starting...
W20260925 14:49:16.446277 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:16 dist-test-slave-z2s5 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:16.655664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35073
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:16.737357 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:16.737572 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:16.737596 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:16.738812 3418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:16.738852 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:16.738866 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:16.738878 3418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:16.738934 3418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:16.740396 3418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:16.740667 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:16.740895 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:16.745792 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:16.745906 3418 server_base.cc:1034] running on GCE node
W20260925 14:49:16.745870 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:16.745800 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:16.746317 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:16.746665 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:16.747807 3418 hybrid_clock.cc:648] HybridClock initialized: now 1790347756747795 us; error 28 us; skew 500 ppm
Sep 25 14:49:16 dist-test-slave-z2s5 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:16.750326 3418 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:16.750811 3418 webserver.cc:460] Webserver started at http://127.0.98.62:38203/ using document root <none> and password file <none>
I20260925 14:49:16.751044 3418 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:16.751111 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:16.751269 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:16.752399 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "6e6ca3af3acb45e8a5b26279403057c8"
format_stamp: "Formatted at 2026-09-25 14:49:16 on dist-test-slave-z2s5"
server_key: "4458ff64b9329214edd35f9f17db3d1c"
server_key_iv: "95b60871eb037996e040b1bae5cac599"
server_key_version: "encryptionkey@0"
I20260925 14:49:16.752769 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "6e6ca3af3acb45e8a5b26279403057c8"
format_stamp: "Formatted at 2026-09-25 14:49:16 on dist-test-slave-z2s5"
server_key: "4458ff64b9329214edd35f9f17db3d1c"
server_key_iv: "95b60871eb037996e040b1bae5cac599"
server_key_version: "encryptionkey@0"
I20260925 14:49:16.754094 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:16.754961 3433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:16.755178 3418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:49:16.755290 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "6e6ca3af3acb45e8a5b26279403057c8"
format_stamp: "Formatted at 2026-09-25 14:49:16 on dist-test-slave-z2s5"
server_key: "4458ff64b9329214edd35f9f17db3d1c"
server_key_iv: "95b60871eb037996e040b1bae5cac599"
server_key_version: "encryptionkey@0"
I20260925 14:49:16.755373 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:16.787680 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:16.789695 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:16.789871 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:16.795204 3418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35073
I20260925 14:49:16.795251 3485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35073 every 8 connection(s)
I20260925 14:49:16.795746 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:16.797008 3486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:16.800405 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8: Bootstrap starting.
I20260925 14:49:16.800695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3418
I20260925 14:49:16.800809 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:16.800983 392 external_mini_cluster.cc:1428] Setting key 6e72d54e9318b83ec7f975b53df11736
I20260925 14:49:16.801153 3486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:16.801462 3486 log.cc:824] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:16.802182 3486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8: No bootstrap required, opened a new log
I20260925 14:49:16.804464 3486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6ca3af3acb45e8a5b26279403057c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35073 } }
I20260925 14:49:16.804646 3486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:16.804683 3486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e6ca3af3acb45e8a5b26279403057c8, State: Initialized, Role: FOLLOWER
I20260925 14:49:16.804823 3486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6ca3af3acb45e8a5b26279403057c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35073 } }
I20260925 14:49:16.804913 3486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:16.804942 3486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:16.804978 3486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:16.805635 3486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6ca3af3acb45e8a5b26279403057c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35073 } }
I20260925 14:49:16.805753 3486 leader_election.cc:302] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e6ca3af3acb45e8a5b26279403057c8; no voters:
I20260925 14:49:16.805933 3486 leader_election.cc:288] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:16.805954 3491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:16.806146 3491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [term 1 LEADER]: Becoming Leader. State: Replica: 6e6ca3af3acb45e8a5b26279403057c8, State: Running, Role: LEADER
I20260925 14:49:16.806259 3486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:16.806699 3491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6ca3af3acb45e8a5b26279403057c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35073 } }
I20260925 14:49:16.807058 3491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e6ca3af3acb45e8a5b26279403057c8. Latest consensus state: current_term: 1 leader_uuid: "6e6ca3af3acb45e8a5b26279403057c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6ca3af3acb45e8a5b26279403057c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35073 } } }
I20260925 14:49:16.807155 3491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:16.807251 3492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e6ca3af3acb45e8a5b26279403057c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e6ca3af3acb45e8a5b26279403057c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35073 } } }
I20260925 14:49:16.807343 3492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8 [sys.catalog]: This master's current role is: LEADER
Sep 25 14:49:16 dist-test-slave-z2s5 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:16.808451 3495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:16.808947 3495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:16.810978 3495 catalog_manager.cc:1347] Generated new cluster ID: 0f5ac3b5214241a3b69b8f10b9a48392
I20260925 14:49:16.811034 3495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:16.812582 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:16.801533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.0.98.62:35073)
0925 14:49:16.801678 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:16.801689 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:16.801730 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:49:16.802220 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:16.802227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:16.802306 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:16.802483 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:16.803376 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:16.804290 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:16.805395 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:16.805878 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:16.806021 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:16.808850 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:16.808854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:16.808855 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:16.808867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:16.810997 (+ 2130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:16.811384 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:16.811388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:16.811388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:16.811485 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:16.811707 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:16.811708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:16.811708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:16.811970 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:16.812097 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:16.812215 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 14:49:16.812366 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260925 14:49:16.821339 3495 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:16.821851 3495 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:16.826452 3495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e6ca3af3acb45e8a5b26279403057c8: Generated new TSK 0
I20260925 14:49:16.826654 3495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:16.855033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35073
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:16.955945 3512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:16.956198 3512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:16.956238 3512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:16.957427 3512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:16.957495 3512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:16.957540 3512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:16.959163 3512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:16.959419 3512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:16.959645 3512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:16.965452 3518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:16.965437 3520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:16.965435 3517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:16.966665 3512 server_base.cc:1034] running on GCE node
I20260925 14:49:16.966897 3512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:16.967222 3512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:16.968434 3512 hybrid_clock.cc:648] HybridClock initialized: now 1790347756968416 us; error 36 us; skew 500 ppm
Sep 25 14:49:16 dist-test-slave-z2s5 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:16.971129 3512 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:16.971618 3512 webserver.cc:460] Webserver started at http://127.0.98.1:37215/ using document root <none> and password file <none>
I20260925 14:49:16.971846 3512 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:16.971911 3512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:16.972064 3512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:16.973122 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "8418657d90524d4c89994560fc0533d0"
format_stamp: "Formatted at 2026-09-25 14:49:16 on dist-test-slave-z2s5"
server_key: "a8b2159af565893239fca022aa4cd26f"
server_key_iv: "de6bb7deadda922d78abd4e45bce33e9"
server_key_version: "encryptionkey@0"
I20260925 14:49:16.973493 3512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8418657d90524d4c89994560fc0533d0"
format_stamp: "Formatted at 2026-09-25 14:49:16 on dist-test-slave-z2s5"
server_key: "a8b2159af565893239fca022aa4cd26f"
server_key_iv: "de6bb7deadda922d78abd4e45bce33e9"
server_key_version: "encryptionkey@0"
I20260925 14:49:16.974889 3512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:16.975641 3527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:16.975804 3512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:16.975904 3512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "8418657d90524d4c89994560fc0533d0"
format_stamp: "Formatted at 2026-09-25 14:49:16 on dist-test-slave-z2s5"
server_key: "a8b2159af565893239fca022aa4cd26f"
server_key_iv: "de6bb7deadda922d78abd4e45bce33e9"
server_key_version: "encryptionkey@0"
I20260925 14:49:16.975975 3512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:16.984509 3512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:16.986788 3512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:16.986943 3512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:16.987169 3512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:16.987455 3512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:16.987506 3512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:16.987537 3512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:16.987566 3512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:16.993624 3512 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38153
I20260925 14:49:16.993665 3640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38153 every 8 connection(s)
I20260925 14:49:16.993973 3512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:49:16 dist-test-slave-z2s5 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:17.000495 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3512
I20260925 14:49:17.000586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:17.000810 392 external_mini_cluster.cc:1428] Setting key 82983fb0df4fa31813d68a088066f845
I20260925 14:49:17.003520 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:16.994716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51089 (local address 127.0.98.62:35073)
0925 14:49:16.994803 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:16.994806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:16.994976 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:49:16.995480 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:16.995481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:16.995491 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:16.995552 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:16.996469 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:16.996963 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:16.997975 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:16.998170 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:16.998227 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:17.000506 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:17.000507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:17.000508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:17.000516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:17.002041 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.002537 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.002538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.002539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.002602 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.002862 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.002863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.002863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.002982 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:17.003070 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:17.003347 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 14:49:17.003428 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:49:17.003743 3643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:16.994674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35073 (local address 127.0.98.1:51089)
0925 14:49:16.994851 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:16.994887 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:16.995358 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:16.995641 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:16.995644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:16.996198 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:16.996385 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:16.996389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:16.997062 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:16.997063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:16.997892 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:16.997894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:16.998265 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:16.998266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:16.998376 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:16.999013 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:16.999026 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:17.000402 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:17.002132 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:17.002139 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:17.002142 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:17.002377 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:17.002672 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:17.002673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:17.002674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:17.002791 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:17.003048 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:17.003055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:17.003206 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 14:49:17.003326 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:49:17.003488 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260925 14:49:17.004238 3641 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35073
I20260925 14:49:17.004341 3641 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:17.004536 3641 heartbeater.cc:511] Master 127.0.98.62:35073 requested a full tablet report, sending...
I20260925 14:49:17.005131 3450 ts_manager.cc:194] Registered new tserver with Master: 8418657d90524d4c89994560fc0533d0 (127.0.98.1:38153)
I20260925 14:49:17.006115 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51089
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:17.025734 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35073
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:17.116684 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:17.116899 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:17.116951 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:17.118201 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:17.118247 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:17.118418 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:17.120247 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:17.120507 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:17.120730 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:17.127748 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:17.127980 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:17.127774 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:17.127889 3646 server_base.cc:1034] running on GCE node
I20260925 14:49:17.128263 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:17.128523 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:17.129686 3646 hybrid_clock.cc:648] HybridClock initialized: now 1790347757129667 us; error 37 us; skew 500 ppm
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:17.132345 3646 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:17.132799 3646 webserver.cc:460] Webserver started at http://127.0.98.2:34419/ using document root <none> and password file <none>
I20260925 14:49:17.133064 3646 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:17.133123 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:17.133262 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:17.134284 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "9049bf528c544cfbb00e4fd0ace780e2"
format_stamp: "Formatted at 2026-09-25 14:49:17 on dist-test-slave-z2s5"
server_key: "9329138cbaaa1ae3b1f96a1e3cfa213f"
server_key_iv: "01c8e05926f49e51d8b318d7540642a7"
server_key_version: "encryptionkey@0"
I20260925 14:49:17.134699 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9049bf528c544cfbb00e4fd0ace780e2"
format_stamp: "Formatted at 2026-09-25 14:49:17 on dist-test-slave-z2s5"
server_key: "9329138cbaaa1ae3b1f96a1e3cfa213f"
server_key_iv: "01c8e05926f49e51d8b318d7540642a7"
server_key_version: "encryptionkey@0"
I20260925 14:49:17.136045 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:17.136921 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:17.137085 3646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:17.137192 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "9049bf528c544cfbb00e4fd0ace780e2"
format_stamp: "Formatted at 2026-09-25 14:49:17 on dist-test-slave-z2s5"
server_key: "9329138cbaaa1ae3b1f96a1e3cfa213f"
server_key_iv: "01c8e05926f49e51d8b318d7540642a7"
server_key_version: "encryptionkey@0"
I20260925 14:49:17.137279 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:17.154955 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:17.156816 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:17.156986 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:17.157236 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:17.157600 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:17.157655 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:17.157698 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:17.157727 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:17.163918 3646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36141
I20260925 14:49:17.163961 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36141 every 8 connection(s)
I20260925 14:49:17.164276 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:17.170610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3646
I20260925 14:49:17.170730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:17.170928 392 external_mini_cluster.cc:1428] Setting key b90339a6908030c99bd3403416d00b15
I20260925 14:49:17.174165 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:17.165045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59887 (local address 127.0.98.62:35073)
0925 14:49:17.165141 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:17.165143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:17.165315 (+ 172us) server_negotiation.cc:407] Connection header received
0925 14:49:17.165788 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:17.165789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:17.165799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:17.165873 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:17.166890 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.167495 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.168534 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.168734 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.168794 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:17.171128 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:17.171130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:17.171130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:17.171139 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:17.172655 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.173163 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.173164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.173164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.173232 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.173504 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.173505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.173505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.173660 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:17.173732 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:17.173990 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 14:49:17.174088 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:49:17.174230 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:17.165009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35073 (local address 127.0.98.2:59887)
0925 14:49:17.165175 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:17.165218 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:17.165699 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:17.165931 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:17.165934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:17.166582 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:17.166780 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:17.166786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.167612 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.167613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:17.168434 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:17.168437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.168812 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.168812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:17.168910 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:17.169400 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:17.169408 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:17.171039 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:17.172741 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:17.172748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:17.172750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:17.173079 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:17.173321 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:17.173323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:17.173323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:17.173433 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:17.173738 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:17.173742 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:17.173861 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:49:17.173949 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:49:17.174071 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260925 14:49:17.174857 3775 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35073
I20260925 14:49:17.174952 3775 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:17.175139 3775 heartbeater.cc:511] Master 127.0.98.62:35073 requested a full tablet report, sending...
I20260925 14:49:17.175580 3450 ts_manager.cc:194] Registered new tserver with Master: 9049bf528c544cfbb00e4fd0ace780e2 (127.0.98.2:36141)
I20260925 14:49:17.175963 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59887
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:17.196604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35073
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:17.277112 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:17.277302 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:17.277324 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:17.278375 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:17.278434 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:17.278460 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:17.279874 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:17.280068 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:17.280232 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:17.285205 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:17.285318 3780 server_base.cc:1034] running on GCE node
W20260925 14:49:17.285245 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:17.285207 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:17.285751 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:17.285944 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:17.287091 3780 hybrid_clock.cc:648] HybridClock initialized: now 1790347757287077 us; error 31 us; skew 500 ppm
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:17.289584 3780 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:17.290045 3780 webserver.cc:460] Webserver started at http://127.0.98.3:37905/ using document root <none> and password file <none>
I20260925 14:49:17.290273 3780 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:17.290318 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:17.290510 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:17.291612 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "9713f5fba48f459b93e3cbedf29412d3"
format_stamp: "Formatted at 2026-09-25 14:49:17 on dist-test-slave-z2s5"
server_key: "f894c7288c3b43d0c48ad69b9c7b598c"
server_key_iv: "f6b8d7ed92b9960feded54d42bb3e2ad"
server_key_version: "encryptionkey@0"
I20260925 14:49:17.291954 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9713f5fba48f459b93e3cbedf29412d3"
format_stamp: "Formatted at 2026-09-25 14:49:17 on dist-test-slave-z2s5"
server_key: "f894c7288c3b43d0c48ad69b9c7b598c"
server_key_iv: "f6b8d7ed92b9960feded54d42bb3e2ad"
server_key_version: "encryptionkey@0"
I20260925 14:49:17.293206 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:17.293915 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:17.294088 3780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:17.294184 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "9713f5fba48f459b93e3cbedf29412d3"
format_stamp: "Formatted at 2026-09-25 14:49:17 on dist-test-slave-z2s5"
server_key: "f894c7288c3b43d0c48ad69b9c7b598c"
server_key_iv: "f6b8d7ed92b9960feded54d42bb3e2ad"
server_key_version: "encryptionkey@0"
I20260925 14:49:17.294267 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:17.320503 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:17.322355 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:17.322547 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:17.322777 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:17.323091 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:17.323145 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:17.323187 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:17.323215 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:17.329015 3780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32961
I20260925 14:49:17.329064 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32961 every 8 connection(s)
I20260925 14:49:17.329366 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:17.331354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3780
I20260925 14:49:17.331457 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:17.331645 392 external_mini_cluster.cc:1428] Setting key d2beed02a61169faeea0fcb1b65173a6
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347757, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:17.338898 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:17.330073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57317 (local address 127.0.98.62:35073)
0925 14:49:17.330115 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:17.330116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:17.330349 (+ 233us) server_negotiation.cc:407] Connection header received
0925 14:49:17.330925 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:17.330927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:17.330940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:17.330997 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:17.332273 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.332768 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.333840 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.334031 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.334111 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:17.336179 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:17.336180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:17.336181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:17.336188 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:17.337514 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.337917 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.337918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.337918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.337978 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.338236 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.338236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.338237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.338346 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:17.338456 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:17.338723 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 14:49:17.338803 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:17.339111 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:17.330065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35073 (local address 127.0.98.3:57317)
0925 14:49:17.330231 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:17.330267 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:17.330811 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:17.331075 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:17.331078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:17.331919 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:17.332201 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:17.332207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.332859 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.332860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:17.333734 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:17.333736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.334120 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.334121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:17.334258 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:17.334799 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:17.334807 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:17.336081 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:17.337609 (+ 1528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:17.337616 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:17.337618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:17.337839 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:17.338042 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:17.338043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:17.338043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:17.338152 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:17.338433 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:17.338439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:17.338587 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 14:49:17.338700 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:49:17.338867 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260925 14:49:17.339593 3909 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35073
I20260925 14:49:17.339684 3909 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:17.339915 3909 heartbeater.cc:511] Master 127.0.98.62:35073 requested a full tablet report, sending...
I20260925 14:49:17.340272 3450 ts_manager.cc:194] Registered new tserver with Master: 9713f5fba48f459b93e3cbedf29412d3 (127.0.98.3:32961)
I20260925 14:49:17.340638 3450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57317
I20260925 14:49:17.343721 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3405](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3405](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347757, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:17.366350 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:17.359276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56696 (local address 127.0.98.62:35073)
0925 14:49:17.359322 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:17.359324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:17.359382 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:49:17.359444 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:17.359444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:17.359453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:17.359505 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:17.360108 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.360583 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.361295 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:17.361466 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:17.361513 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:17.363683 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:17.363684 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:17.363684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:17.363697 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:17.365335 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.365620 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.365621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.365621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.365675 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:17.365826 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:17.365827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:17.365827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:17.365966 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:17.366091 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:17.366131 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 14:49:17.366192 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:49:17.366678 3450 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56696
I20260925 14:49:17.367194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3512
I20260925 14:49:17.372262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3646
I20260925 14:49:17.376999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3780
I20260925 14:49:17.381129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3418
2026-09-25T14:49:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1981 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3921](info): set up 2 sockets
Sep 25 14:49:17 dist-test-slave-z2s5 krb5kdc[3921](info): commencing operation
krb5kdc: starting...
W20260925 14:49:18.422231 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:18 dist-test-slave-z2s5 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:18.538024 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45787
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:18.616463 3934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:18.616710 3934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:18.616746 3934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:18.617878 3934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:18.617952 3934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:18.617976 3934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:18.617998 3934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:18.618027 3934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:18.619470 3934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45787
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:18.619745 3934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:18.619962 3934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:18.624886 3939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:18.624890 3940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:18.624995 3942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:18.625114 3934 server_base.cc:1034] running on GCE node
I20260925 14:49:18.625406 3934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:18.625677 3934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:18.626833 3934 hybrid_clock.cc:648] HybridClock initialized: now 1790347758626821 us; error 44 us; skew 500 ppm
Sep 25 14:49:18 dist-test-slave-z2s5 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:18.633455 3934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:18.633873 3934 webserver.cc:460] Webserver started at http://127.0.98.62:35255/ using document root <none> and password file <none>
I20260925 14:49:18.634099 3934 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:18.634163 3934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:18.634313 3934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:18.635396 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "acb954c089ca4765b89731755658ec7f"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "c782f193d8ab00796cdd40124e929cfe"
server_key_iv: "30245aea5da1c84d5363e529e398c3f1"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.635752 3934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "acb954c089ca4765b89731755658ec7f"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "c782f193d8ab00796cdd40124e929cfe"
server_key_iv: "30245aea5da1c84d5363e529e398c3f1"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.637526 3934 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 14:49:18.638230 3949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:18.638439 3934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:18.638545 3934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "acb954c089ca4765b89731755658ec7f"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "c782f193d8ab00796cdd40124e929cfe"
server_key_iv: "30245aea5da1c84d5363e529e398c3f1"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.638624 3934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:18.672482 3934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:18.674938 3934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:18.675098 3934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:18.678887 3934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45787
I20260925 14:49:18.678946 4001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45787 every 8 connection(s)
I20260925 14:49:18.679298 3934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:18.680181 4002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:18.682364 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f: Bootstrap starting.
I20260925 14:49:18.683035 4002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:18.683082 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 3934
I20260925 14:49:18.683176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:18.683390 392 external_mini_cluster.cc:1428] Setting key eda8dbb9f2812a5346f76a3864b8b6d4
I20260925 14:49:18.683380 4002 log.cc:824] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:18.684178 4002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f: No bootstrap required, opened a new log
I20260925 14:49:18.686830 4002 raft_consensus.cc:348] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb954c089ca4765b89731755658ec7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45787 } }
I20260925 14:49:18.687027 4002 raft_consensus.cc:374] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:18.687065 4002 raft_consensus.cc:729] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acb954c089ca4765b89731755658ec7f, State: Initialized, Role: FOLLOWER
I20260925 14:49:18.687189 4002 consensus_queue.cc:260] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb954c089ca4765b89731755658ec7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45787 } }
I20260925 14:49:18.687260 4002 raft_consensus.cc:388] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:18.687294 4002 raft_consensus.cc:482] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:18.687335 4002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:18.687942 4002 raft_consensus.cc:504] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb954c089ca4765b89731755658ec7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45787 } }
I20260925 14:49:18.688028 4002 leader_election.cc:302] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acb954c089ca4765b89731755658ec7f; no voters:
I20260925 14:49:18.688210 4002 leader_election.cc:288] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:18.688237 4007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:18.688390 4007 raft_consensus.cc:686] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [term 1 LEADER]: Becoming Leader. State: Replica: acb954c089ca4765b89731755658ec7f, State: Running, Role: LEADER
I20260925 14:49:18.688467 4002 sys_catalog.cc:564] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:18.688525 4007 consensus_queue.cc:237] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb954c089ca4765b89731755658ec7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45787 } }
I20260925 14:49:18.688891 4008 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acb954c089ca4765b89731755658ec7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb954c089ca4765b89731755658ec7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45787 } } }
I20260925 14:49:18.688879 4007 sys_catalog.cc:455] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader acb954c089ca4765b89731755658ec7f. Latest consensus state: current_term: 1 leader_uuid: "acb954c089ca4765b89731755658ec7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acb954c089ca4765b89731755658ec7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45787 } } }
I20260925 14:49:18.688969 4008 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:18.688977 4007 sys_catalog.cc:458] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:18.690002 4018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 14:49:18 dist-test-slave-z2s5 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:18.690608 4018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:18.694101 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:18.683877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34292 (local address 127.0.98.62:45787)
0925 14:49:18.684019 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:18.684027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:18.684129 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:49:18.684644 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:18.684650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:18.684697 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:18.684812 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:18.685596 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:18.686552 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:18.687595 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:18.687983 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:18.688127 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:18.691026 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:18.691030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:18.691048 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:18.691061 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:18.692682 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:18.693096 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:18.693098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:18.693099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:18.693181 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:18.693392 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:18.693392 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:18.693393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:18.693600 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:18.693725 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:18.693857 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:49:18.693952 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260925 14:49:18.695428 4018 catalog_manager.cc:1347] Generated new cluster ID: ad8856194fb54cf188e62c97365a43c6
I20260925 14:49:18.695497 4018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:18.708525 4018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:18.709079 4018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:18.713701 4018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acb954c089ca4765b89731755658ec7f: Generated new TSK 0
I20260925 14:49:18.713876 4018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:18.742640 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45787
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:18.823320 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:18.823516 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:18.823539 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:18.824638 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:18.824676 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:18.824702 4028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:18.826059 4028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:18.826243 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:18.826450 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:18.831316 4036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:18.831346 4034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:18.831316 4033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:18.831588 4028 server_base.cc:1034] running on GCE node
I20260925 14:49:18.831812 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:18.832056 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:18.833216 4028 hybrid_clock.cc:648] HybridClock initialized: now 1790347758833208 us; error 42 us; skew 500 ppm
Sep 25 14:49:18 dist-test-slave-z2s5 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:18.835819 4028 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:18.836199 4028 webserver.cc:460] Webserver started at http://127.0.98.1:38395/ using document root <none> and password file <none>
I20260925 14:49:18.836375 4028 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:18.836416 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:18.836503 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:18.837468 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "5b87e74723b14ee2b4c9077d472a38d1"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "e4b099a824036639731a368c4dc3557c"
server_key_iv: "573843b3fb47f14da9d9528d5c9950fc"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.837764 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5b87e74723b14ee2b4c9077d472a38d1"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "e4b099a824036639731a368c4dc3557c"
server_key_iv: "573843b3fb47f14da9d9528d5c9950fc"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.839080 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:18.839728 4043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:18.839892 4028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:18.839967 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "5b87e74723b14ee2b4c9077d472a38d1"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "e4b099a824036639731a368c4dc3557c"
server_key_iv: "573843b3fb47f14da9d9528d5c9950fc"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.840018 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:18.856634 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:18.858613 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:18.858726 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:18.858898 4028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:18.859195 4028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:18.859228 4028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:18.859257 4028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:18.859306 4028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:18.864840 4028 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45623
I20260925 14:49:18.864923 4156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45623 every 8 connection(s)
I20260925 14:49:18.865258 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:18.867301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4028
I20260925 14:49:18.867417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:18.867592 392 external_mini_cluster.cc:1428] Setting key ce9ab3820e294c1359301ca667e97f56
Sep 25 14:49:18 dist-test-slave-z2s5 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:49:18.876031 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:18.866062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58073 (local address 127.0.98.62:45787)
0925 14:49:18.866129 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:18.866131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:18.866343 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:49:18.867310 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:18.867312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:18.867325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:18.867406 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:18.868243 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:18.869038 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:18.870461 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:18.870661 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:18.870716 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:18.873123 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:18.873124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:18.873124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:18.873139 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:18.874608 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:18.875022 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:18.875024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:18.875024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:18.875083 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:18.875337 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:18.875338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:18.875338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:18.875455 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:18.875534 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:18.875855 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 14:49:18.875940 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:49:18.876211 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:18.866033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45787 (local address 127.0.98.1:58073)
0925 14:49:18.866203 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:18.866240 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:18.867190 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:18.867498 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:18.867503 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:18.867879 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:18.868147 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:18.868154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:18.869129 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:18.869130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:18.870334 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:18.870337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:18.870745 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:18.870747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:18.870909 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:18.871424 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:18.871435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:18.873012 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:18.874698 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:18.874705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:18.874707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:18.874931 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:18.875151 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:18.875152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:18.875153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:18.875267 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:18.875524 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:18.875530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:18.875701 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 14:49:18.875833 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 14:49:18.875985 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260925 14:49:18.876700 4157 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45787
I20260925 14:49:18.876796 4157 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:18.876938 4157 heartbeater.cc:511] Master 127.0.98.62:45787 requested a full tablet report, sending...
I20260925 14:49:18.877413 3966 ts_manager.cc:194] Registered new tserver with Master: 5b87e74723b14ee2b4c9077d472a38d1 (127.0.98.1:45623)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 14:49:18.878018 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:18.893356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45787
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:18.973193 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:18.973448 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:18.973486 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:18.974670 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:18.974745 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:18.974790 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:18.976300 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:18.976562 4162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:18.976788 4162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:18.981660 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:18.981643 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:18.981643 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:18.982020 4162 server_base.cc:1034] running on GCE node
I20260925 14:49:18.982239 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:18.982518 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:18.983678 4162 hybrid_clock.cc:648] HybridClock initialized: now 1790347758983667 us; error 29 us; skew 500 ppm
Sep 25 14:49:18 dist-test-slave-z2s5 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:18.986137 4162 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:18.986668 4162 webserver.cc:460] Webserver started at http://127.0.98.2:34949/ using document root <none> and password file <none>
I20260925 14:49:18.986865 4162 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:18.986905 4162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:18.986990 4162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:18.987991 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "ed188679bb2d454faf5cb6a9ebde6fcb"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "efe7bd15f06da3b726df57a86c5c9bf4"
server_key_iv: "b8b2535927f30652a39fbc2068a849ff"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.988358 4162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ed188679bb2d454faf5cb6a9ebde6fcb"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "efe7bd15f06da3b726df57a86c5c9bf4"
server_key_iv: "b8b2535927f30652a39fbc2068a849ff"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.989835 4162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:18.990561 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:18.990772 4162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:18.990844 4162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "ed188679bb2d454faf5cb6a9ebde6fcb"
format_stamp: "Formatted at 2026-09-25 14:49:18 on dist-test-slave-z2s5"
server_key: "efe7bd15f06da3b726df57a86c5c9bf4"
server_key_iv: "b8b2535927f30652a39fbc2068a849ff"
server_key_version: "encryptionkey@0"
I20260925 14:49:18.990895 4162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:18.999620 4162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:19.001305 4162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:19.001412 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:19.001574 4162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:19.001806 4162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:19.001835 4162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:19.001856 4162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:19.001868 4162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:19.007417 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41279
I20260925 14:49:19.007490 4290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41279 every 8 connection(s)
I20260925 14:49:19.007798 4162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:19.008047 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4162
I20260925 14:49:19.008154 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:19.008360 392 external_mini_cluster.cc:1428] Setting key c5cd973fda47899d0cf57d824676b1de
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 14:49:19.017864 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:19.008522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52795 (local address 127.0.98.62:45787)
0925 14:49:19.008572 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:19.008574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:19.009082 (+ 508us) server_negotiation.cc:407] Connection header received
0925 14:49:19.010003 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:19.010004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:19.010018 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:19.010088 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:19.010795 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.011292 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.012300 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.012509 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.012596 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:19.015119 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:19.015120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:19.015121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:19.015127 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:19.016627 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.016994 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.016995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.016995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.017045 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.017332 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.017333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.017333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.017442 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:19.017497 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:19.017726 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 14:49:19.017798 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:49:19.018105 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:19.008521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45787 (local address 127.0.98.2:52795)
0925 14:49:19.008919 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:19.008976 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:19.009893 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:19.010178 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:19.010181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:19.010499 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:19.010687 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:19.010692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.011384 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.011385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:19.012182 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:19.012184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.012604 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.012605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:19.012729 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:19.013288 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:19.013297 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:19.015024 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:19.016704 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:19.016710 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:19.016712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:19.016928 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:19.017153 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:19.017154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:19.017154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:19.017267 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:19.017503 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:19.017507 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:19.017619 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 14:49:19.017702 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:49:19.017839 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":96,"threads_started":1}
I20260925 14:49:19.018682 4291 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45787
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260925 14:49:19.018800 4291 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:19.019003 4291 heartbeater.cc:511] Master 127.0.98.62:45787 requested a full tablet report, sending...
I20260925 14:49:19.019474 3966 ts_manager.cc:194] Registered new tserver with Master: ed188679bb2d454faf5cb6a9ebde6fcb (127.0.98.2:41279)
I20260925 14:49:19.019997 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:19.033524 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45787
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:19.112458 4296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:19.112712 4296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:19.112751 4296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:19.113903 4296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:19.113987 4296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:19.114032 4296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:19.115536 4296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45787
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:19.115819 4296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:19.116050 4296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:19.121117 4301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:19.121083 4302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:19.121357 4296 server_base.cc:1034] running on GCE node
W20260925 14:49:19.121083 4304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:19.121625 4296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:19.121865 4296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:19.123019 4296 hybrid_clock.cc:648] HybridClock initialized: now 1790347759123013 us; error 31 us; skew 500 ppm
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:19.125459 4296 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:19.125880 4296 webserver.cc:460] Webserver started at http://127.0.98.3:33315/ using document root <none> and password file <none>
I20260925 14:49:19.126097 4296 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:19.126163 4296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:19.126310 4296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:19.127368 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "aad3d8679434449a9d5cc1efd4240ccf"
format_stamp: "Formatted at 2026-09-25 14:49:19 on dist-test-slave-z2s5"
server_key: "88b2f44390ae114485f0a2b25e9749a7"
server_key_iv: "d1d04c45934fdf2ce766c25e6354f668"
server_key_version: "encryptionkey@0"
I20260925 14:49:19.127724 4296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aad3d8679434449a9d5cc1efd4240ccf"
format_stamp: "Formatted at 2026-09-25 14:49:19 on dist-test-slave-z2s5"
server_key: "88b2f44390ae114485f0a2b25e9749a7"
server_key_iv: "d1d04c45934fdf2ce766c25e6354f668"
server_key_version: "encryptionkey@0"
I20260925 14:49:19.129009 4296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:19.129699 4311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:19.129863 4296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:19.129957 4296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "aad3d8679434449a9d5cc1efd4240ccf"
format_stamp: "Formatted at 2026-09-25 14:49:19 on dist-test-slave-z2s5"
server_key: "88b2f44390ae114485f0a2b25e9749a7"
server_key_iv: "d1d04c45934fdf2ce766c25e6354f668"
server_key_version: "encryptionkey@0"
I20260925 14:49:19.130033 4296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:19.142957 4296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:19.144748 4296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:19.144910 4296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:19.145135 4296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:19.145438 4296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:19.145489 4296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:19.145531 4296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:19.145560 4296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:19.151399 4296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43823
I20260925 14:49:19.151449 4424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43823 every 8 connection(s)
I20260925 14:49:19.151799 4296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:19.158149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4296
I20260925 14:49:19.158255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:19.158500 392 external_mini_cluster.cc:1428] Setting key a298de69ba843b6eafda889874bd638d
I20260925 14:49:19.161798 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:19.152495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42935 (local address 127.0.98.62:45787)
0925 14:49:19.152564 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:19.152566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:19.152787 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:49:19.153667 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:19.153668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:19.153678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:19.153730 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:19.154288 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.154848 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.156310 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.156508 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.156566 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:19.158873 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:19.158874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:19.158874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:19.158899 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:19.160424 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.160846 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.160847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.160848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.160911 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.161164 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.161165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.161166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.161288 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:19.161362 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:19.161631 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:49:19.161706 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:49:19.161999 4427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:19.152485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45787 (local address 127.0.98.3:42935)
0925 14:49:19.152656 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:19.152691 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:19.153581 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:19.153786 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:19.153789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:19.154035 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:19.154219 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:19.154222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.154923 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.154924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:19.156210 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:19.156213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.156579 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.156580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:19.156676 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:19.157211 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:19.157219 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:19.158763 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:19.160525 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:19.160532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:19.160534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:19.160759 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:19.160977 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:19.160977 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:19.160978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:19.161089 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:19.161354 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:19.161359 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:19.161496 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 14:49:19.161610 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:49:19.161772 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260925 14:49:19.162477 4425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45787
I20260925 14:49:19.162585 4425 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:19.162762 4425 heartbeater.cc:511] Master 127.0.98.62:45787 requested a full tablet report, sending...
I20260925 14:49:19.163121 3966 ts_manager.cc:194] Registered new tserver with Master: aad3d8679434449a9d5cc1efd4240ccf (127.0.98.3:43823)
I20260925 14:49:19.163450 3966 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42935
I20260925 14:49:19.170274 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[3921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:19.192044 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:19.184398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42918 (local address 127.0.98.1:45623)
0925 14:49:19.184562 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:19.184567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:19.184577 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:19.184601 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:19.184603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:19.184616 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:19.184679 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:19.185185 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.185922 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.186688 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.187024 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.187076 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:19.189381 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:19.189400 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:19.189401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:19.189407 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:19.191056 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.191362 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.191365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.191365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.191427 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.191585 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.191586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.191586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.191705 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:19.191814 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:19.191866 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:49:19.191924 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[3921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:19.199882 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:19.193191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34298 (local address 127.0.98.62:45787)
0925 14:49:19.193246 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:19.193248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:19.193277 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:49:19.193376 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:19.193377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:19.193388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:19.193451 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:19.193936 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.194600 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.195297 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:19.195501 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:19.195581 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:19.197571 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:19.197572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:19.197573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:19.197580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:19.198737 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.199083 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.199085 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.199085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.199143 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:19.199338 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:19.199339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:19.199339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:19.199484 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:19.199589 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:19.199693 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 14:49:19.199780 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 14:49:19.200127 3966 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34298
I20260925 14:49:19.200500 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4028
I20260925 14:49:19.204898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4162
I20260925 14:49:19.208814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4296
I20260925 14:49:19.212858 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 3934
2026-09-25T14:49:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1831 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[4437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[4437](info): set up 2 sockets
Sep 25 14:49:19 dist-test-slave-z2s5 krb5kdc[4437](info): commencing operation
krb5kdc: starting...
W20260925 14:49:20.246371 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:20.358856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42463
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:20.440757 4450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:20.440963 4450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:20.440985 4450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:20.442147 4450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:20.442190 4450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:20.442204 4450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:20.442216 4450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:20.442273 4450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:20.443790 4450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:20.444072 4450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:20.444314 4450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:20.449035 4455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.449060 4458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.449082 4456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:20.449335 4450 server_base.cc:1034] running on GCE node
I20260925 14:49:20.449580 4450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:20.449860 4450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:20.451046 4450 hybrid_clock.cc:648] HybridClock initialized: now 1790347760451039 us; error 35 us; skew 500 ppm
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:20.453401 4450 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:20.453737 4450 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:20.453820 4450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:20.453969 4450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:20.455097 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "4343cd7c2cbe4005b323c32340cb3386"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "e6629cd73acd2eec4ff9ba5cbfc44e73"
server_key_iv: "7496fedd63c9d1fba49548aa8f5ae837"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.455453 4450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "4343cd7c2cbe4005b323c32340cb3386"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "e6629cd73acd2eec4ff9ba5cbfc44e73"
server_key_iv: "7496fedd63c9d1fba49548aa8f5ae837"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.456740 4450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:20.457430 4464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.457612 4450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:20.457715 4450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "4343cd7c2cbe4005b323c32340cb3386"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "e6629cd73acd2eec4ff9ba5cbfc44e73"
server_key_iv: "7496fedd63c9d1fba49548aa8f5ae837"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.457800 4450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:20.480170 4450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:20.482579 4450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:20.482745 4450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:20.486405 4450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42463
I20260925 14:49:20.486476 4516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42463 every 8 connection(s)
I20260925 14:49:20.486774 4450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:20.487677 4517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:20.489924 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386: Bootstrap starting.
I20260925 14:49:20.490530 4517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:20.490828 4517 log.cc:824] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:20.491482 4517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386: No bootstrap required, opened a new log
I20260925 14:49:20.492997 4517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4343cd7c2cbe4005b323c32340cb3386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260925 14:49:20.493132 4517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:20.493201 4517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4343cd7c2cbe4005b323c32340cb3386, State: Initialized, Role: FOLLOWER
I20260925 14:49:20.493357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4450
I20260925 14:49:20.493372 4517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4343cd7c2cbe4005b323c32340cb3386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260925 14:49:20.493458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:20.493481 4517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:20.493526 4517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:20.493573 4517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:20.493645 392 external_mini_cluster.cc:1428] Setting key cc48b6fd10e704c665d3907695ee6459
I20260925 14:49:20.494743 4517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4343cd7c2cbe4005b323c32340cb3386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260925 14:49:20.494848 4517 leader_election.cc:302] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4343cd7c2cbe4005b323c32340cb3386; no voters:
I20260925 14:49:20.495008 4517 leader_election.cc:288] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:20.495038 4522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:20.495141 4522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [term 1 LEADER]: Becoming Leader. State: Replica: 4343cd7c2cbe4005b323c32340cb3386, State: Running, Role: LEADER
I20260925 14:49:20.495265 4517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:20.495282 4522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4343cd7c2cbe4005b323c32340cb3386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } }
I20260925 14:49:20.495754 4524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4343cd7c2cbe4005b323c32340cb3386. Latest consensus state: current_term: 1 leader_uuid: "4343cd7c2cbe4005b323c32340cb3386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4343cd7c2cbe4005b323c32340cb3386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } } }
I20260925 14:49:20.495862 4524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:20.496296 4530 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:20.496438 4523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4343cd7c2cbe4005b323c32340cb3386" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4343cd7c2cbe4005b323c32340cb3386" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42463 } } }
I20260925 14:49:20.496501 4523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:20.496838 4530 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:20.498370 4530 catalog_manager.cc:1347] Generated new cluster ID: f163642cbf0749d3ad04a9a021bc2167
I20260925 14:49:20.498443 4530 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:20.505200 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:20.494148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58014 (local address 127.0.98.62:42463)
0925 14:49:20.494375 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:20.494409 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:20.494452 (+ 43us) server_negotiation.cc:407] Connection header received
0925 14:49:20.494570 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:20.494577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:20.494633 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:20.494775 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:20.495572 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.497008 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.498084 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.498692 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.498803 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:20.501616 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:20.501619 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:20.501620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:20.501631 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:20.503553 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:20.503995 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:20.503998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:20.503998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:20.504113 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:20.504339 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:20.504339 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:20.504340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:20.504616 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:20.504756 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:20.504888 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:49:20.505014 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260925 14:49:20.516185 4530 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:20.516707 4530 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:20.521435 4530 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386: Generated new TSK 0
I20260925 14:49:20.521629 4530 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:20.552764 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:20.655536 4543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:20.655730 4543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:20.655752 4543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:20.656905 4543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:20.656946 4543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:20.656972 4543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:20.658571 4543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:20.658805 4543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:20.659111 4543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:20.664104 4548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.664273 4551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.664300 4549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:20.664459 4543 server_base.cc:1034] running on GCE node
I20260925 14:49:20.664682 4543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:20.664923 4543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:20.666038 4543 hybrid_clock.cc:648] HybridClock initialized: now 1790347760666034 us; error 40 us; skew 500 ppm
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:20.668834 4543 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:20.669224 4543 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:20.669306 4543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:20.669447 4543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:20.670555 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "e7865bba0a154f7c88c4bf2ade11e0c0"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "42cce744ef52d9d9954cd06a919b1977"
server_key_iv: "83aff4dead1393dacb77989ab3e4eb0d"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.670935 4543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e7865bba0a154f7c88c4bf2ade11e0c0"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "42cce744ef52d9d9954cd06a919b1977"
server_key_iv: "83aff4dead1393dacb77989ab3e4eb0d"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.672188 4543 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:20.672844 4557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.673123 4543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:20.673218 4543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "e7865bba0a154f7c88c4bf2ade11e0c0"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "42cce744ef52d9d9954cd06a919b1977"
server_key_iv: "83aff4dead1393dacb77989ab3e4eb0d"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.673300 4543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:20.687999 4543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:20.689787 4543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:20.689952 4543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:20.690169 4543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:20.690573 4543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:20.690627 4543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.690668 4543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:20.690696 4543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.696916 4543 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38241
I20260925 14:49:20.696956 4670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38241 every 8 connection(s)
I20260925 14:49:20.697238 4543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:20.698000 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4543
I20260925 14:49:20.698144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:20.698411 392 external_mini_cluster.cc:1428] Setting key 68e6cd6ec578f3f3bf66fa40bbb1335d
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:49:20.708235 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:20.698031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49123 (local address 127.0.98.62:42463)
0925 14:49:20.698109 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:20.698111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:20.698373 (+ 262us) server_negotiation.cc:407] Connection header received
0925 14:49:20.699069 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:20.699070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:20.699081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:20.699149 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:20.700148 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.700735 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.702158 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.702376 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.702495 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:20.705029 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:20.705031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:20.705031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:20.705049 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:20.706700 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:20.707155 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:20.707156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:20.707156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:20.707222 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:20.707500 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:20.707501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:20.707501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:20.707626 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:20.707714 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:20.708035 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 14:49:20.708138 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:49:20.708437 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:20.698008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.1:49123)
0925 14:49:20.698217 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:20.698272 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:20.698952 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:20.699232 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:20.699237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:20.699526 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:20.700038 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:20.700043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.700841 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.700842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:20.702059 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:20.702061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.702562 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.702564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:20.702725 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:20.703240 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:20.703250 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:20.704897 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:20.706803 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:20.706810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:20.706813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:20.707047 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:20.707299 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:20.707300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:20.707301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:20.707422 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:20.707702 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:20.707708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:20.707860 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 14:49:20.708007 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 14:49:20.708178 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 14:49:20.708973 4671 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42463
I20260925 14:49:20.709084 4671 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:20.709271 4671 heartbeater.cc:511] Master 127.0.98.62:42463 requested a full tablet report, sending...
I20260925 14:49:20.709789 4481 ts_manager.cc:194] Registered new tserver with Master: e7865bba0a154f7c88c4bf2ade11e0c0 (127.0.98.1:38241)
I20260925 14:49:20.710424 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:20.724750 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:20.819767 4676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:20.819971 4676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:20.819995 4676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:20.821074 4676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:20.821112 4676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:20.821138 4676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:20.822644 4676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:20.822839 4676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:20.823081 4676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:20.828092 4684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.828135 4681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:20.828316 4676 server_base.cc:1034] running on GCE node
W20260925 14:49:20.828359 4682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:20.828557 4676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:20.828745 4676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:20.829922 4676 hybrid_clock.cc:648] HybridClock initialized: now 1790347760829915 us; error 42 us; skew 500 ppm
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:20.832420 4676 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:20.832816 4676 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:20.832890 4676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:20.833042 4676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:20.834087 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "2deaab3f396b48668313acb5188ac253"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "4f0a0d824caf7eaba8b7499fd15ed01b"
server_key_iv: "d3f7b0cdd825be91b12770bfdf7927ba"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.834532 4676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2deaab3f396b48668313acb5188ac253"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "4f0a0d824caf7eaba8b7499fd15ed01b"
server_key_iv: "d3f7b0cdd825be91b12770bfdf7927ba"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.835860 4676 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:20.836582 4690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.836808 4676 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:20.836905 4676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "2deaab3f396b48668313acb5188ac253"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "4f0a0d824caf7eaba8b7499fd15ed01b"
server_key_iv: "d3f7b0cdd825be91b12770bfdf7927ba"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.836993 4676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:20.858016 4676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:20.860406 4676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:20.860570 4676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:20.860813 4676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:20.861199 4676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:20.861253 4676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.861287 4676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:20.861315 4676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.867339 4676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38209
I20260925 14:49:20.867393 4803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38209 every 8 connection(s)
I20260925 14:49:20.867683 4676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:20.873590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4676
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:20.873723 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:20.873919 392 external_mini_cluster.cc:1428] Setting key 652027a866855481829d63b5fb74fa31
I20260925 14:49:20.877516 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:20.868511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58047 (local address 127.0.98.62:42463)
0925 14:49:20.868592 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:20.868595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:20.868803 (+ 208us) server_negotiation.cc:407] Connection header received
0925 14:49:20.869260 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:20.869260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:20.869271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:20.869339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:20.870240 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.870808 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.871792 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.872003 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.872074 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:20.874285 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:20.874287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:20.874287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:20.874294 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:20.875984 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:20.876554 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:20.876555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:20.876556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:20.876606 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:20.876884 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:20.876884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:20.876884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:20.876984 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:20.877067 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:20.877379 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 14:49:20.877447 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:49:20.877832 4806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:20.868491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.2:58047)
0925 14:49:20.868669 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:20.868712 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:20.869177 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:20.869396 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:20.869400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:20.869661 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:20.870158 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:20.870163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.870890 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.870891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:20.871693 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:20.871695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:20.872085 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:20.872086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:20.872182 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:20.872710 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:20.872719 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:20.874176 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:20.876076 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:20.876087 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:20.876089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:20.876479 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:20.876675 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:20.876676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:20.876676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:20.876818 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:20.877060 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:20.877066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:20.877243 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:49:20.877362 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 14:49:20.877541 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260925 14:49:20.878322 4804 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42463
I20260925 14:49:20.878459 4804 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:20.878636 4804 heartbeater.cc:511] Master 127.0.98.62:42463 requested a full tablet report, sending...
I20260925 14:49:20.879001 4481 ts_manager.cc:194] Registered new tserver with Master: 2deaab3f396b48668313acb5188ac253 (127.0.98.2:38209)
I20260925 14:49:20.879328 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58047
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:20.901631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:20.981971 4809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:20.982210 4809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:20.982249 4809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:20.983531 4809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:20.983623 4809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:20.983670 4809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:20.985200 4809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:20.985466 4809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:20.985693 4809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:20.990653 4817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.990633 4815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:20.990643 4814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:20.991006 4809 server_base.cc:1034] running on GCE node
I20260925 14:49:20.991241 4809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:20.991493 4809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:20.992655 4809 hybrid_clock.cc:648] HybridClock initialized: now 1790347760992646 us; error 47 us; skew 500 ppm
Sep 25 14:49:20 dist-test-slave-z2s5 krb5kdc[4437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:20.995028 4809 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:20.995358 4809 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:20.995428 4809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:20.995594 4809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:20.996757 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac748e9ee73e456898b86458ec49a40c"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "84bf6a5f5d61020b721a0985c18ad85d"
server_key_iv: "78f6ae4a9c43d105bedfdd0a13f64a7a"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.997138 4809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac748e9ee73e456898b86458ec49a40c"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "84bf6a5f5d61020b721a0985c18ad85d"
server_key_iv: "78f6ae4a9c43d105bedfdd0a13f64a7a"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.998373 4809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:20.999109 4823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:20.999285 4809 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:20.999380 4809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "ac748e9ee73e456898b86458ec49a40c"
format_stamp: "Formatted at 2026-09-25 14:49:20 on dist-test-slave-z2s5"
server_key: "84bf6a5f5d61020b721a0985c18ad85d"
server_key_iv: "78f6ae4a9c43d105bedfdd0a13f64a7a"
server_key_version: "encryptionkey@0"
I20260925 14:49:20.999460 4809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:21.013372 4809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:21.015383 4809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:21.015544 4809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:21.015774 4809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:21.016131 4809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:21.016189 4809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.016230 4809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:21.016256 4809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.022081 4809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43781
I20260925 14:49:21.022119 4936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43781 every 8 connection(s)
I20260925 14:49:21.022460 4809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:21.026203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4809
I20260925 14:49:21.026326 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:21.026520 392 external_mini_cluster.cc:1428] Setting key ae954075774b2821583023afeba0f277
Sep 25 14:49:21 dist-test-slave-z2s5 krb5kdc[4437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:21.031422 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.023134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57441 (local address 127.0.98.62:42463)
0925 14:49:21.023178 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.023180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.023435 (+ 255us) server_negotiation.cc:407] Connection header received
0925 14:49:21.023963 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.023964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.023975 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.024034 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.025007 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.025512 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.026602 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.026834 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.026903 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.029096 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.029097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.029098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:21.029104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.030258 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:21.030670 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:21.030671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:21.030671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:21.030722 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:21.030929 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:21.030930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:21.030930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:21.031029 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.031078 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.031298 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.031361 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:21.031636 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.023118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42463 (local address 127.0.98.3:57441)
0925 14:49:21.023314 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.023349 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.023855 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.024106 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.024110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.024396 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.024898 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.024903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.025607 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.025608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.026505 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.026507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.026988 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.026989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.027143 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.027685 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:21.027701 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.029016 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.030327 (+ 1311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:21.030332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:21.030333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:21.030607 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:21.030785 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:21.030785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:21.030786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:21.030878 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:21.031079 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.031083 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.031192 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.031273 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.031386 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260925 14:49:21.032142 4937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42463
I20260925 14:49:21.032258 4937 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:21.032413 4937 heartbeater.cc:511] Master 127.0.98.62:42463 requested a full tablet report, sending...
I20260925 14:49:21.032778 4481 ts_manager.cc:194] Registered new tserver with Master: ac748e9ee73e456898b86458ec49a40c (127.0.98.3:43781)
I20260925 14:49:21.033149 4481 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57441
I20260925 14:49:21.038213 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:49:21.045297 4521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.039472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.0.98.62:42463)
0925 14:49:21.039509 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.039511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.039687 (+ 176us) server_negotiation.cc:407] Connection header received
0925 14:49:21.039746 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.039747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.039757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.039819 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.040363 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.040862 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.041574 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.041766 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.041855 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.043023 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.043024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.043025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:21.043031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.044138 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:21.044497 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:21.044498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:21.044499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:21.044558 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:21.044752 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:21.044753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:21.044753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:21.044910 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.045020 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.045119 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.045197 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:21.046779 4481 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:21.048219 4481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:21.054291 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.050306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38209 (local address 127.0.0.1:56232)
0925 14:49:21.050504 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.050550 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.050614 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.050956 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.050959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.050961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:21.051023 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.051025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.051964 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.051965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.053066 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.053068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.053957 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.053958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.054043 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.054057 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.054155 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.054212 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260925 14:49:21.054296 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.050485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56232 (local address 127.0.98.2:38209)
0925 14:49:21.050751 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.050756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.050783 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:49:21.050823 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.050825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.050841 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.050926 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:21.051109 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.051866 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.053161 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.053869 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.053956 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.054017 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.054176 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.054217 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":77,"threads_started":1}
I20260925 14:49:21.054567 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.050345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43781 (local address 127.0.0.1:45272)
0925 14:49:21.050705 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.050714 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.050753 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.051136 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.051136 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.051139 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:21.051203 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.051205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.052516 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.052517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.053425 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.053427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.054324 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.054325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.054410 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.054415 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.054449 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.054490 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":24,"thread_start_us":89,"threads_started":1}
I20260925 14:49:21.054770 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.050514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.0.1:36672)
0925 14:49:21.050834 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.050843 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.050880 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.051140 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.051141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.051142 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:21.051203 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.051205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.052504 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.052505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.053618 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.053620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.054560 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.054561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.054618 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.054621 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.054673 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.054707 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":55,"thread_start_us":66,"threads_started":1}
I20260925 14:49:21.054919 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.050457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45272 (local address 127.0.98.3:43781)
0925 14:49:21.050978 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.050982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.050990 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:21.051025 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.051027 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.051042 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.051278 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:21.051353 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.052386 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.053520 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.054223 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.054681 (+ 458us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.054740 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.054765 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.054824 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":84,"threads_started":1}
I20260925 14:49:21.055230 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.050723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36672 (local address 127.0.98.1:38241)
0925 14:49:21.050897 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.050903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.050927 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:49:21.050966 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.050968 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.050984 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.051060 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:21.051284 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.052392 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.053700 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.054420 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.054479 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.054531 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.054908 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.054953 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260925 14:49:21.055478 4871 tablet_service.cc:1467] Processing CreateTablet for tablet f2318a8d4ce14ad09bd87bd7f626d32d (DEFAULT_TABLE table=test-table [id=dfd4294d059c43b2b71cdd2910661e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:21.055444 4738 tablet_service.cc:1467] Processing CreateTablet for tablet f2318a8d4ce14ad09bd87bd7f626d32d (DEFAULT_TABLE table=test-table [id=dfd4294d059c43b2b71cdd2910661e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:21.055787 4871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2318a8d4ce14ad09bd87bd7f626d32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.055794 4738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2318a8d4ce14ad09bd87bd7f626d32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.055770 4605 tablet_service.cc:1467] Processing CreateTablet for tablet f2318a8d4ce14ad09bd87bd7f626d32d (DEFAULT_TABLE table=test-table [id=dfd4294d059c43b2b71cdd2910661e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:21.056008 4605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2318a8d4ce14ad09bd87bd7f626d32d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.059026 4955 tablet_bootstrap.cc:492] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: Bootstrap starting.
I20260925 14:49:21.059095 4954 tablet_bootstrap.cc:492] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: Bootstrap starting.
I20260925 14:49:21.059448 4956 tablet_bootstrap.cc:492] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Bootstrap starting.
I20260925 14:49:21.059701 4955 tablet_bootstrap.cc:654] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.059835 4954 tablet_bootstrap.cc:654] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.060302 4956 tablet_bootstrap.cc:654] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.060317 4955 log.cc:824] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.060504 4954 log.cc:824] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.060930 4956 log.cc:824] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.061228 4955 tablet_bootstrap.cc:492] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: No bootstrap required, opened a new log
I20260925 14:49:21.061288 4955 ts_tablet_manager.cc:1397] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:21.061461 4954 tablet_bootstrap.cc:492] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: No bootstrap required, opened a new log
I20260925 14:49:21.061517 4954 ts_tablet_manager.cc:1397] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:21.061578 4956 tablet_bootstrap.cc:492] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: No bootstrap required, opened a new log
I20260925 14:49:21.061630 4956 ts_tablet_manager.cc:1397] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:21.062958 4955 raft_consensus.cc:348] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.063112 4955 raft_consensus.cc:374] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.063154 4955 raft_consensus.cc:729] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2deaab3f396b48668313acb5188ac253, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.063284 4955 consensus_queue.cc:260] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.063565 4955 ts_tablet_manager.cc:1428] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:49:21.063524 4956 raft_consensus.cc:348] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.063524 4954 raft_consensus.cc:348] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.063678 4954 raft_consensus.cc:374] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.063678 4956 raft_consensus.cc:374] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.063701 4804 heartbeater.cc:503] Master 127.0.98.62:42463 was elected leader, sending a full tablet report...
I20260925 14:49:21.063714 4954 raft_consensus.cc:729] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7865bba0a154f7c88c4bf2ade11e0c0, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.063714 4956 raft_consensus.cc:729] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac748e9ee73e456898b86458ec49a40c, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.063848 4954 consensus_queue.cc:260] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.063841 4956 consensus_queue.cc:260] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.064098 4954 ts_tablet_manager.cc:1428] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:21.064098 4956 ts_tablet_manager.cc:1428] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:49:21.064217 4671 heartbeater.cc:503] Master 127.0.98.62:42463 was elected leader, sending a full tablet report...
I20260925 14:49:21.064253 4937 heartbeater.cc:503] Master 127.0.98.62:42463 was elected leader, sending a full tablet report...
W20260925 14:49:21.118292 4805 tablet.cc:2369] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:49:21.197898 4672 tablet.cc:2369] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:21.208189 4961 raft_consensus.cc:482] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:21.208331 4961 raft_consensus.cc:504] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.208658 4961 leader_election.cc:288] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2deaab3f396b48668313acb5188ac253 (127.0.98.2:38209), e7865bba0a154f7c88c4bf2ade11e0c0 (127.0.98.1:38241)
I20260925 14:49:21.211601 4951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.208927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44573 (local address 127.0.98.2:38209)
0925 14:49:21.208993 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.208995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.209006 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:21.209038 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.209039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.209051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.209125 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:21.209328 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.209853 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.210872 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.211249 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.211304 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.211335 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.211491 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.211531 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:49:21.211612 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.209082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58285 (local address 127.0.98.1:38241)
0925 14:49:21.209124 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.209126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.209219 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:49:21.209251 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.209252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.209271 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.209325 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:21.209456 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.209880 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.210921 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.211252 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.211350 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.211359 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.211519 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.211554 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:21.211604 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.208851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38209 (local address 127.0.98.3:44573)
0925 14:49:21.208917 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.208930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.208976 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.209138 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.209139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.209141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:21.209244 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.209246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.209939 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.209940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.210785 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.210787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.211332 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.211333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.211416 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.211419 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.211462 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.211520 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":4}
I20260925 14:49:21.211618 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.208976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.98.3:58285)
0925 14:49:21.209137 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.209148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.209212 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.209327 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.209328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.209329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:21.209390 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.209391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.209990 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.209990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.210817 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:21.210819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.211363 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.211364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:21.211425 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.211428 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.211501 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.211535 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260925 14:49:21.211994 4758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2318a8d4ce14ad09bd87bd7f626d32d" candidate_uuid: "ac748e9ee73e456898b86458ec49a40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2deaab3f396b48668313acb5188ac253" is_pre_election: true
I20260925 14:49:21.212157 4758 raft_consensus.cc:2454] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac748e9ee73e456898b86458ec49a40c in term 0.
I20260925 14:49:21.212201 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2318a8d4ce14ad09bd87bd7f626d32d" candidate_uuid: "ac748e9ee73e456898b86458ec49a40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" is_pre_election: true
I20260925 14:49:21.212342 4625 raft_consensus.cc:2454] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac748e9ee73e456898b86458ec49a40c in term 0.
I20260925 14:49:21.212543 4825 leader_election.cc:302] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac748e9ee73e456898b86458ec49a40c, e7865bba0a154f7c88c4bf2ade11e0c0; no voters:
I20260925 14:49:21.212698 4961 raft_consensus.cc:2781] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:21.212776 4961 raft_consensus.cc:482] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:21.212803 4961 raft_consensus.cc:3037] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.213477 4961 raft_consensus.cc:504] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.213605 4961 leader_election.cc:288] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [CANDIDATE]: Term 1 election: Requested vote from peers 2deaab3f396b48668313acb5188ac253 (127.0.98.2:38209), e7865bba0a154f7c88c4bf2ade11e0c0 (127.0.98.1:38241)
I20260925 14:49:21.213766 4758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2318a8d4ce14ad09bd87bd7f626d32d" candidate_uuid: "ac748e9ee73e456898b86458ec49a40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2deaab3f396b48668313acb5188ac253"
I20260925 14:49:21.213794 4625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2318a8d4ce14ad09bd87bd7f626d32d" candidate_uuid: "ac748e9ee73e456898b86458ec49a40c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0"
I20260925 14:49:21.213840 4758 raft_consensus.cc:3037] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.213886 4625 raft_consensus.cc:3037] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.214591 4625 raft_consensus.cc:2454] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac748e9ee73e456898b86458ec49a40c in term 1.
I20260925 14:49:21.214623 4758 raft_consensus.cc:2454] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac748e9ee73e456898b86458ec49a40c in term 1.
I20260925 14:49:21.214789 4825 leader_election.cc:302] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2deaab3f396b48668313acb5188ac253, ac748e9ee73e456898b86458ec49a40c; no voters:
I20260925 14:49:21.214926 4961 raft_consensus.cc:2781] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:21.215059 4961 raft_consensus.cc:686] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 1 LEADER]: Becoming Leader. State: Replica: ac748e9ee73e456898b86458ec49a40c, State: Running, Role: LEADER
I20260925 14:49:21.215161 4961 consensus_queue.cc:237] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } }
I20260925 14:49:21.215997 4480 catalog_manager.cc:5495] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c reported cstate change: term changed from 0 to 1, leader changed from <none> to ac748e9ee73e456898b86458ec49a40c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ac748e9ee73e456898b86458ec49a40c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac748e9ee73e456898b86458ec49a40c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43781 } health_report { overall_health: HEALTHY } } }
I20260925 14:49:21.234666 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.232102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45282 (local address 127.0.98.3:43781)
0925 14:49:21.232139 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.232142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.232151 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:21.232245 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.232246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.232264 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.232362 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:21.232516 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.233031 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.233801 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:21.233999 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:21.234087 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:21.234252 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:21.234367 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:21.234512 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.234531 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.234551 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:49:21.236418 4625 raft_consensus.cc:1261] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 1 FOLLOWER]: Refusing update from remote peer ac748e9ee73e456898b86458ec49a40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:21.236429 4758 raft_consensus.cc:1261] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 1 FOLLOWER]: Refusing update from remote peer ac748e9ee73e456898b86458ec49a40c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:21.236822 4964 consensus_queue.cc:1035] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2deaab3f396b48668313acb5188ac253" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:21.236960 4961 consensus_queue.cc:1035] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7865bba0a154f7c88c4bf2ade11e0c0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:21.239979 4968 mvcc.cc:205] Tried to move back new op lower bound from 7333264430021791744 to 7333264429937905664. Current Snapshot: MvccSnapshot[applied={T|T < 7333264430021791744}]
I20260925 14:49:21.241557 4480 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:21.241671 4480 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:21.242378 4480 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4343cd7c2cbe4005b323c32340cb3386: Sending DeleteTablet for 3 replicas of tablet f2318a8d4ce14ad09bd87bd7f626d32d
I20260925 14:49:21.242710 4738 tablet_service.cc:1514] Processing DeleteTablet for tablet f2318a8d4ce14ad09bd87bd7f626d32d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56232
I20260925 14:49:21.242748 4871 tablet_service.cc:1514] Processing DeleteTablet for tablet f2318a8d4ce14ad09bd87bd7f626d32d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45272
I20260925 14:49:21.242944 4605 tablet_service.cc:1514] Processing DeleteTablet for tablet f2318a8d4ce14ad09bd87bd7f626d32d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36672
I20260925 14:49:21.242986 4977 tablet_replica.cc:330] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: stopping tablet replica
I20260925 14:49:21.243093 4977 raft_consensus.cc:2229] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:21.243188 4978 tablet_replica.cc:330] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0: stopping tablet replica
I20260925 14:49:21.243196 4977 raft_consensus.cc:2258] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:21.243471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4543
I20260925 14:49:21.243413 4978 raft_consensus.cc:2229] T f2318a8d4ce14ad09bd87bd7f626d32d P e7865bba0a154f7c88c4bf2ade11e0c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:21.244534 4977 ts_tablet_manager.cc:1905] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:21.246138 4977 ts_tablet_manager.cc:1918] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:21.246212 4977 log.cc:1196] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/f2318a8d4ce14ad09bd87bd7f626d32d
I20260925 14:49:21.246649 4977 ts_tablet_manager.cc:1939] T f2318a8d4ce14ad09bd87bd7f626d32d P ac748e9ee73e456898b86458ec49a40c: Deleting consensus metadata
I20260925 14:49:21.247049 4465 catalog_manager.cc:4841] TS ac748e9ee73e456898b86458ec49a40c (127.0.98.3:43781): tablet f2318a8d4ce14ad09bd87bd7f626d32d (table test-table [id=dfd4294d059c43b2b71cdd2910661e8b]) successfully deleted
W20260925 14:49:21.250469 4466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38241 (error 108)
I20260925 14:49:21.250660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4676
I20260925 14:49:21.250695 4976 tablet_replica.cc:330] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253: stopping tablet replica
I20260925 14:49:21.250823 4976 raft_consensus.cc:2229] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:21.250929 4976 raft_consensus.cc:2258] T f2318a8d4ce14ad09bd87bd7f626d32d P 2deaab3f396b48668313acb5188ac253 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:21.250988 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.250879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38241 (local address 127.0.0.1:36682)
0925 14:49:21.250922 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.250950 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 14:49:21.251767 4466 catalog_manager.cc:4584] TS e7865bba0a154f7c88c4bf2ade11e0c0 (127.0.98.1:38241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2318a8d4ce14ad09bd87bd7f626d32d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38241: connect: Connection refused (error 111)
I20260925 14:49:21.255571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4809
I20260925 14:49:21.255590 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.255494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38209 (local address 127.0.0.1:56234)
0925 14:49:21.255538 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.255560 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:49:21.259335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4450
2026-09-25T14:49:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2046 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:21Z Disabled control of system clock
I20260925 14:49:21.285953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46547
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:49:21.365127 4983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:21.365337 4983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:21.365358 4983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:21.366580 4983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:21.366621 4983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:21.366633 4983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:21.366647 4983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:21.366657 4983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:21.368006 4983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46547
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:21.368248 4983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:21.368541 4983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:21.373535 4988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.373569 4989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.373543 4991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:21.373734 4983 server_base.cc:1034] running on GCE node
I20260925 14:49:21.373965 4983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:21.374254 4983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:21.375413 4983 hybrid_clock.cc:648] HybridClock initialized: now 1790347761375419 us; error 61 us; skew 500 ppm
I20260925 14:49:21.376582 4983 webserver.cc:460] Webserver started at http://127.0.98.62:40687/ using document root <none> and password file <none>
I20260925 14:49:21.376816 4983 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:21.376863 4983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:21.377012 4983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:21.378067 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "cdce463646da43f5b0040c0727d545be"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "8ad8fbfd12b0f7aabd8d82e31f9a3ebc"
server_key_iv: "3d2488914ce5704c228776b3af696eca"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.378489 4983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "cdce463646da43f5b0040c0727d545be"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "8ad8fbfd12b0f7aabd8d82e31f9a3ebc"
server_key_iv: "3d2488914ce5704c228776b3af696eca"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.379725 4983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:21.380404 4997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.380590 4983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:21.380658 4983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "cdce463646da43f5b0040c0727d545be"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "8ad8fbfd12b0f7aabd8d82e31f9a3ebc"
server_key_iv: "3d2488914ce5704c228776b3af696eca"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.380743 4983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:21.384083 4983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:21.384353 4983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:21.384498 4983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:21.388443 4983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46547
I20260925 14:49:21.388473 5049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46547 every 8 connection(s)
I20260925 14:49:21.388794 4983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:21.389395 5050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.390133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 4983
I20260925 14:49:21.390255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:21.390493 392 external_mini_cluster.cc:1428] Setting key a0f2d1d7389add8097a7a8c935b01496
I20260925 14:49:21.392920 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be: Bootstrap starting.
I20260925 14:49:21.392977 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.391033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.0.98.62:46547)
0925 14:49:21.391244 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.391252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.391279 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:49:21.391682 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.391688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.391726 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.391852 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.392525 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.392529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.392529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.392532 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.392569 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.392579 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.392633 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.392724 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.392808 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
I20260925 14:49:21.393522 5050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.393813 5050 log.cc:824] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.394524 5050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be: No bootstrap required, opened a new log
I20260925 14:49:21.395818 5050 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdce463646da43f5b0040c0727d545be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260925 14:49:21.395978 5050 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.396024 5050 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdce463646da43f5b0040c0727d545be, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.396175 5050 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdce463646da43f5b0040c0727d545be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260925 14:49:21.396283 5050 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:21.396328 5050 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:21.396378 5050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.397238 5050 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdce463646da43f5b0040c0727d545be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260925 14:49:21.397365 5050 leader_election.cc:302] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdce463646da43f5b0040c0727d545be; no voters:
I20260925 14:49:21.397557 5050 leader_election.cc:288] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:21.397593 5055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:21.397785 5055 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [term 1 LEADER]: Becoming Leader. State: Replica: cdce463646da43f5b0040c0727d545be, State: Running, Role: LEADER
I20260925 14:49:21.397848 5050 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:21.397884 5055 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdce463646da43f5b0040c0727d545be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } }
I20260925 14:49:21.398483 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdce463646da43f5b0040c0727d545be. Latest consensus state: current_term: 1 leader_uuid: "cdce463646da43f5b0040c0727d545be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdce463646da43f5b0040c0727d545be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } } }
I20260925 14:49:21.398501 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdce463646da43f5b0040c0727d545be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdce463646da43f5b0040c0727d545be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46547 } } }
I20260925 14:49:21.398581 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:21.398607 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:21.398782 5064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:21.399277 5064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:21.400794 5064 catalog_manager.cc:1347] Generated new cluster ID: 70f4f4dca8064fddbaba6e13c6c654b5
I20260925 14:49:21.400851 5064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:21.413259 5064 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:21.413470 5064 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:21.416504 5064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be: Generated new TSK 0
I20260925 14:49:21.416688 5064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:21.419149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:49:21.499892 5074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:21.500121 5074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:21.500159 5074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:21.501370 5074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:21.501436 5074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:21.501479 5074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:21.502933 5074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:21.503182 5074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:21.503469 5074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:21.509222 5082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.509218 5080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.509303 5079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:21.509558 5074 server_base.cc:1034] running on GCE node
I20260925 14:49:21.509756 5074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:21.510017 5074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:21.511160 5074 hybrid_clock.cc:648] HybridClock initialized: now 1790347761511142 us; error 35 us; skew 500 ppm
I20260925 14:49:21.512488 5074 webserver.cc:460] Webserver started at http://127.0.98.1:43471/ using document root <none> and password file <none>
I20260925 14:49:21.512735 5074 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:21.512782 5074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:21.512923 5074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:21.513981 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "48c41a1044904ed1adc033d619ddcc8b"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "f7394d7e837b204f0b005878da68ac0b"
server_key_iv: "4d89f2c521a6d52f57bcd9409b83cb98"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.514343 5074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48c41a1044904ed1adc033d619ddcc8b"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "f7394d7e837b204f0b005878da68ac0b"
server_key_iv: "4d89f2c521a6d52f57bcd9409b83cb98"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.515724 5074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:21.516469 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.516664 5074 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:21.516788 5074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "48c41a1044904ed1adc033d619ddcc8b"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "f7394d7e837b204f0b005878da68ac0b"
server_key_iv: "4d89f2c521a6d52f57bcd9409b83cb98"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.516891 5074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:21.521463 5074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:21.521737 5074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:21.521862 5074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:21.522054 5074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:21.522343 5074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:21.522382 5074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.522430 5074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:21.522456 5074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.529266 5074 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42787
I20260925 14:49:21.529340 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42787 every 8 connection(s)
I20260925 14:49:21.529675 5074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:21.532485 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.530378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58461 (local address 127.0.98.62:46547)
0925 14:49:21.530451 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.530454 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.530755 (+ 301us) server_negotiation.cc:407] Connection header received
0925 14:49:21.531188 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.531189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.531194 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.531247 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.532283 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.532284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.532284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.532288 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.532301 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.532305 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.532355 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.532422 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.532434 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:49:21.532671 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.530376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46547 (local address 127.0.98.1:58461)
0925 14:49:21.530596 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.530654 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.531105 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.531302 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.531306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.531314 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.532143 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.532143 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.532170 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.532197 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.532355 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.532357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.532359 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.532410 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.532454 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260925 14:49:21.533177 5202 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46547
I20260925 14:49:21.533250 5202 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:21.533416 5202 heartbeater.cc:511] Master 127.0.98.62:46547 requested a full tablet report, sending...
I20260925 14:49:21.533890 5014 ts_manager.cc:194] Registered new tserver with Master: 48c41a1044904ed1adc033d619ddcc8b (127.0.98.1:42787)
I20260925 14:49:21.535107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5074
I20260925 14:49:21.535218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:21.535384 392 external_mini_cluster.cc:1428] Setting key dd136754a9510a65212a7252f0428621
I20260925 14:49:21.536214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:49:21.614866 5205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:21.615114 5205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:21.615151 5205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:21.616308 5205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:21.616381 5205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:21.616425 5205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:21.617761 5205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:21.618037 5205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:21.618326 5205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:21.623497 5205 server_base.cc:1034] running on GCE node
W20260925 14:49:21.623593 5213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.623647 5211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.623679 5210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:21.623961 5205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:21.624217 5205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:21.625384 5205 hybrid_clock.cc:648] HybridClock initialized: now 1790347761625382 us; error 38 us; skew 500 ppm
I20260925 14:49:21.626605 5205 webserver.cc:460] Webserver started at http://127.0.98.2:38125/ using document root <none> and password file <none>
I20260925 14:49:21.626852 5205 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:21.626919 5205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:21.627063 5205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:21.628101 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "be1dee3ca50746e697d6e3b29d65d3fa"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "80d7e5c8e55f05c57e0a211aad154279"
server_key_iv: "1d745e4b227edfe3ed00b4660ed4ce40"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.628455 5205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be1dee3ca50746e697d6e3b29d65d3fa"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "80d7e5c8e55f05c57e0a211aad154279"
server_key_iv: "1d745e4b227edfe3ed00b4660ed4ce40"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.629863 5205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:21.630609 5219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.630781 5205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:21.630851 5205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "be1dee3ca50746e697d6e3b29d65d3fa"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "80d7e5c8e55f05c57e0a211aad154279"
server_key_iv: "1d745e4b227edfe3ed00b4660ed4ce40"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.630908 5205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:21.634052 5205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:21.634272 5205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:21.634361 5205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:21.634606 5205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:21.634857 5205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:21.634888 5205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.634907 5205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:21.634919 5205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.640524 5205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40195
I20260925 14:49:21.640642 5332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40195 every 8 connection(s)
I20260925 14:49:21.640928 5205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:21.644201 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.641682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53837 (local address 127.0.98.62:46547)
0925 14:49:21.641817 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.641819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.642036 (+ 217us) server_negotiation.cc:407] Connection header received
0925 14:49:21.642688 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.642690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.642698 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.642776 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.643956 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.643957 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.643957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.643962 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.643977 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.643981 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.644040 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.644117 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.644132 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 14:49:21.644394 5335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.641673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46547 (local address 127.0.98.2:53837)
0925 14:49:21.641897 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.641940 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.642539 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.642846 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.642850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.642860 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.643754 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.643755 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.643784 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.643821 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.644041 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.644046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.644049 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.644128 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.644188 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260925 14:49:21.644860 5333 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46547
I20260925 14:49:21.644932 5333 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:21.645035 5333 heartbeater.cc:511] Master 127.0.98.62:46547 requested a full tablet report, sending...
I20260925 14:49:21.645342 5014 ts_manager.cc:194] Registered new tserver with Master: be1dee3ca50746e697d6e3b29d65d3fa (127.0.98.2:40195)
I20260925 14:49:21.650846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5205
I20260925 14:49:21.650954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:21.651149 392 external_mini_cluster.cc:1428] Setting key aafdcfe2cf752fef54200b30873f6853
I20260925 14:49:21.652076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--builtin_ntp_servers=127.0.98.20:45267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:49:21.732678 5336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:21.732910 5336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:21.732950 5336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:21.734155 5336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:21.734221 5336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:21.734265 5336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:21.735627 5336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:21.735900 5336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:21.736189 5336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:21.741106 5342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:21.741215 5344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:21.741343 5336 server_base.cc:1034] running on GCE node
W20260925 14:49:21.741353 5341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:21.741657 5336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:21.741863 5336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:21.743024 5336 hybrid_clock.cc:648] HybridClock initialized: now 1790347761743012 us; error 44 us; skew 500 ppm
I20260925 14:49:21.744354 5336 webserver.cc:460] Webserver started at http://127.0.98.3:36415/ using document root <none> and password file <none>
I20260925 14:49:21.744590 5336 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:21.744663 5336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:21.744818 5336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:21.745862 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "157a322da53b4215832644e2284319b7"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "cee30833a125b43c00fde0a8a0d16eea"
server_key_iv: "a4757c684cc00fe3b32dd1e672bcffb9"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.746234 5336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "157a322da53b4215832644e2284319b7"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "cee30833a125b43c00fde0a8a0d16eea"
server_key_iv: "a4757c684cc00fe3b32dd1e672bcffb9"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.747566 5336 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:21.748327 5350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.748498 5336 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:21.748596 5336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "157a322da53b4215832644e2284319b7"
format_stamp: "Formatted at 2026-09-25 14:49:21 on dist-test-slave-z2s5"
server_key: "cee30833a125b43c00fde0a8a0d16eea"
server_key_iv: "a4757c684cc00fe3b32dd1e672bcffb9"
server_key_version: "encryptionkey@0"
I20260925 14:49:21.748679 5336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:21.752030 5336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:21.752292 5336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:21.752434 5336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:21.752646 5336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:21.752946 5336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:21.753001 5336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.753041 5336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:21.753068 5336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:21.758579 5336 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41885
I20260925 14:49:21.758610 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41885 every 8 connection(s)
I20260925 14:49:21.758936 5336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:21.762023 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.759689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59001 (local address 127.0.98.62:46547)
0925 14:49:21.759773 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.759776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.759985 (+ 209us) server_negotiation.cc:407] Connection header received
0925 14:49:21.760499 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.760500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.760507 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.760554 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.761757 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.761758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.761759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.761764 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.761778 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.761782 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.761839 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.761940 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.761955 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:49:21.762217 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.759651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46547 (local address 127.0.98.3:59001)
0925 14:49:21.759867 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.759898 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.760380 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.760624 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.760627 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.760637 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.761604 (+ 967us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.761605 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.761633 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.761659 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.761849 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.761854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.761856 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.761951 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.762011 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260925 14:49:21.762723 5464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46547
I20260925 14:49:21.762815 5464 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:21.762919 5464 heartbeater.cc:511] Master 127.0.98.62:46547 requested a full tablet report, sending...
I20260925 14:49:21.763294 5014 ts_manager.cc:194] Registered new tserver with Master: 157a322da53b4215832644e2284319b7 (127.0.98.3:41885)
I20260925 14:49:21.766536 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5336
I20260925 14:49:21.766636 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:21.766837 392 external_mini_cluster.cc:1428] Setting key e4c922198b0f9e162ad7ca828afb44c0
I20260925 14:49:21.767606 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:49:21.769891 5053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.768862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.98.62:46547)
0925 14:49:21.768921 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.768923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.769016 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:49:21.769053 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.769054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.769058 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.769104 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.769687 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.769688 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.769688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.769691 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.769701 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.769703 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.769742 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.769773 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.769789 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:49:21.771366 5014 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:21.772936 5014 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:21.776604 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.774992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42787 (local address 127.0.0.1:46102)
0925 14:49:21.775152 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.775162 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.775206 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.775463 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.775468 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.775475 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.776173 (+ 698us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.776174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.776183 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.776197 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.776449 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.776450 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.776451 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.776511 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.776524 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260925 14:49:21.776618 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.775087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.0.98.1:42787)
0925 14:49:21.775319 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.775323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.775336 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:49:21.775357 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.775359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.775366 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.775447 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.776287 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.776297 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.776297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.776302 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.776338 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.776365 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.776451 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.776521 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.776537 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260925 14:49:21.776716 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.775342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.0.98.2:40195)
0925 14:49:21.775490 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.775495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.775505 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:21.775535 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.775537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.775546 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.775797 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.776264 (+ 467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.776268 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.776268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.776272 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.776306 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.776335 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.776576 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.776600 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.776615 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260925 14:49:21.776779 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.775193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41885 (local address 127.0.0.1:36540)
0925 14:49:21.775301 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.775310 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.775361 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.776030 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.776031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.776033 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.776419 (+ 386us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.776420 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.776425 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.776435 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.776673 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.776674 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.776674 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.776701 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.776714 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":74,"threads_started":1}
I20260925 14:49:21.777009 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.775578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.0.98.3:41885)
0925 14:49:21.775878 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.775882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.775891 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:21.775914 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.775916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.775925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.776181 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.776554 (+ 373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.776557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.776557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.776560 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.776589 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.776602 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.776884 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.776909 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.776923 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":93,"threads_started":1}
I20260925 14:49:21.777243 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.774982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40195 (local address 127.0.0.1:42550)
0925 14:49:21.775092 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.775132 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.775181 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.775650 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.775652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.775655 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.776038 (+ 383us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.776039 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.776055 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.776077 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.776418 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.776424 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.776426 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.776473 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.776487 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":69,"threads_started":1}
I20260925 14:49:21.777506 5136 tablet_service.cc:1467] Processing CreateTablet for tablet c5e74ada7ad1490994749656644be6fb (DEFAULT_TABLE table=test-table [id=6a18a6a4064449258a2148a5021893f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:21.777653 5398 tablet_service.cc:1467] Processing CreateTablet for tablet c5e74ada7ad1490994749656644be6fb (DEFAULT_TABLE table=test-table [id=6a18a6a4064449258a2148a5021893f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:21.777892 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e74ada7ad1490994749656644be6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.777953 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e74ada7ad1490994749656644be6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.777895 5267 tablet_service.cc:1467] Processing CreateTablet for tablet c5e74ada7ad1490994749656644be6fb (DEFAULT_TABLE table=test-table [id=6a18a6a4064449258a2148a5021893f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:21.778175 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5e74ada7ad1490994749656644be6fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:21.780580 5478 tablet_bootstrap.cc:492] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b: Bootstrap starting.
I20260925 14:49:21.781097 5479 tablet_bootstrap.cc:492] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: Bootstrap starting.
I20260925 14:49:21.781303 5478 tablet_bootstrap.cc:654] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.781631 5478 log.cc:824] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.781723 5479 tablet_bootstrap.cc:654] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.782024 5479 log.cc:824] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.782413 5478 tablet_bootstrap.cc:492] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b: No bootstrap required, opened a new log
I20260925 14:49:21.782363 5480 tablet_bootstrap.cc:492] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Bootstrap starting.
I20260925 14:49:21.782473 5478 ts_tablet_manager.cc:1397] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:21.782742 5479 tablet_bootstrap.cc:492] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: No bootstrap required, opened a new log
I20260925 14:49:21.782821 5479 ts_tablet_manager.cc:1397] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:21.783041 5480 tablet_bootstrap.cc:654] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:21.783344 5480 log.cc:824] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:21.784135 5480 tablet_bootstrap.cc:492] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: No bootstrap required, opened a new log
I20260925 14:49:21.784204 5480 ts_tablet_manager.cc:1397] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:21.785161 5479 raft_consensus.cc:348] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.785310 5479 raft_consensus.cc:374] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.785360 5479 raft_consensus.cc:729] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 157a322da53b4215832644e2284319b7, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.785481 5479 consensus_queue.cc:260] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.785416 5478 raft_consensus.cc:348] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.785591 5478 raft_consensus.cc:374] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.785632 5478 raft_consensus.cc:729] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c41a1044904ed1adc033d619ddcc8b, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.785761 5479 ts_tablet_manager.cc:1428] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:21.785753 5478 consensus_queue.cc:260] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.785940 5464 heartbeater.cc:503] Master 127.0.98.62:46547 was elected leader, sending a full tablet report...
I20260925 14:49:21.786017 5478 ts_tablet_manager.cc:1428] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 14:49:21.786046 5202 heartbeater.cc:503] Master 127.0.98.62:46547 was elected leader, sending a full tablet report...
I20260925 14:49:21.787185 5480 raft_consensus.cc:348] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.787315 5480 raft_consensus.cc:374] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:21.787335 5480 raft_consensus.cc:729] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1dee3ca50746e697d6e3b29d65d3fa, State: Initialized, Role: FOLLOWER
I20260925 14:49:21.787472 5480 consensus_queue.cc:260] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.787721 5480 ts_tablet_manager.cc:1428] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:49:21.787750 5333 heartbeater.cc:503] Master 127.0.98.62:46547 was elected leader, sending a full tablet report...
I20260925 14:49:21.865620 5485 raft_consensus.cc:482] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:21.865759 5485 raft_consensus.cc:504] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.866178 5485 leader_election.cc:288] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be1dee3ca50746e697d6e3b29d65d3fa (127.0.98.2:40195), 157a322da53b4215832644e2284319b7 (127.0.98.3:41885)
I20260925 14:49:21.867784 5204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.866454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40195 (local address 127.0.98.1:37795)
0925 14:49:21.866499 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.866514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.866576 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.866798 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.866799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.866802 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.867405 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.867406 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.867412 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.867424 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.867645 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.867646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.867647 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.867701 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.867715 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 14:49:21.867808 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.866530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37795 (local address 127.0.98.2:40195)
0925 14:49:21.866573 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.866576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.866585 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:21.866674 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.866675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.866680 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.866730 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.867515 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.867516 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.867517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.867522 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.867557 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.867562 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.867646 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.867703 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.867718 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:21.867961 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.866572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41885 (local address 127.0.98.1:56007)
0925 14:49:21.866718 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.866728 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:21.866779 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:21.867003 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:21.867004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:21.867007 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:21.867582 (+ 575us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:21.867582 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:21.867588 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:21.867635 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:21.867812 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:21.867813 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:21.867813 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 14:49:21.867862 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:49:21.867874 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260925 14:49:21.867955 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.866684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56007 (local address 127.0.98.3:41885)
0925 14:49:21.866727 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.866729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.866773 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:49:21.866913 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.866915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.866920 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.866982 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.867719 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.867720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.867721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.867724 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.867738 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.867742 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.867812 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.867864 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.867876 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:49:21.868409 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e74ada7ad1490994749656644be6fb" candidate_uuid: "48c41a1044904ed1adc033d619ddcc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157a322da53b4215832644e2284319b7" is_pre_election: true
I20260925 14:49:21.868443 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e74ada7ad1490994749656644be6fb" candidate_uuid: "48c41a1044904ed1adc033d619ddcc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" is_pre_election: true
I20260925 14:49:21.868570 5418 raft_consensus.cc:2454] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c41a1044904ed1adc033d619ddcc8b in term 0.
I20260925 14:49:21.868592 5287 raft_consensus.cc:2454] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48c41a1044904ed1adc033d619ddcc8b in term 0.
I20260925 14:49:21.868825 5089 leader_election.cc:302] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c41a1044904ed1adc033d619ddcc8b, be1dee3ca50746e697d6e3b29d65d3fa; no voters:
I20260925 14:49:21.868950 5485 raft_consensus.cc:2781] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:21.869007 5485 raft_consensus.cc:482] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:21.869038 5485 raft_consensus.cc:3037] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.869732 5485 raft_consensus.cc:504] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.869871 5485 leader_election.cc:288] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [CANDIDATE]: Term 1 election: Requested vote from peers be1dee3ca50746e697d6e3b29d65d3fa (127.0.98.2:40195), 157a322da53b4215832644e2284319b7 (127.0.98.3:41885)
I20260925 14:49:21.870007 5287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e74ada7ad1490994749656644be6fb" candidate_uuid: "48c41a1044904ed1adc033d619ddcc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be1dee3ca50746e697d6e3b29d65d3fa"
I20260925 14:49:21.870047 5418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5e74ada7ad1490994749656644be6fb" candidate_uuid: "48c41a1044904ed1adc033d619ddcc8b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "157a322da53b4215832644e2284319b7"
I20260925 14:49:21.870090 5287 raft_consensus.cc:3037] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.870150 5418 raft_consensus.cc:3037] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:21.870903 5287 raft_consensus.cc:2454] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c41a1044904ed1adc033d619ddcc8b in term 1.
I20260925 14:49:21.870932 5418 raft_consensus.cc:2454] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48c41a1044904ed1adc033d619ddcc8b in term 1.
I20260925 14:49:21.871140 5089 leader_election.cc:302] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 157a322da53b4215832644e2284319b7, 48c41a1044904ed1adc033d619ddcc8b; no voters:
I20260925 14:49:21.871234 5485 raft_consensus.cc:2781] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:21.871397 5485 raft_consensus.cc:686] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [term 1 LEADER]: Becoming Leader. State: Replica: 48c41a1044904ed1adc033d619ddcc8b, State: Running, Role: LEADER
I20260925 14:49:21.871501 5485 consensus_queue.cc:237] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } }
I20260925 14:49:21.872344 5014 catalog_manager.cc:5495] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b reported cstate change: term changed from 0 to 1, leader changed from <none> to 48c41a1044904ed1adc033d619ddcc8b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "48c41a1044904ed1adc033d619ddcc8b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48c41a1044904ed1adc033d619ddcc8b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 } health_report { overall_health: UNKNOWN } } }
W20260925 14:49:21.891456 5334 tablet.cc:2369] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:21.893258 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.892140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.0.98.1:42787)
0925 14:49:21.892200 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:21.892202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:21.892210 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:21.892274 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:21.892275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:21.892295 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:21.892376 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:21.892999 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:21.893000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:21.893001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:21.893006 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:21.893021 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:21.893025 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:21.893111 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:21.893168 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 14:49:21.893183 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:49:21.895179 5418 raft_consensus.cc:1261] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 1 FOLLOWER]: Refusing update from remote peer 48c41a1044904ed1adc033d619ddcc8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:21.895179 5287 raft_consensus.cc:1261] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 1 FOLLOWER]: Refusing update from remote peer 48c41a1044904ed1adc033d619ddcc8b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:21.895524 5485 consensus_queue.cc:1035] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "157a322da53b4215832644e2284319b7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:21.895687 5488 consensus_queue.cc:1035] T c5e74ada7ad1490994749656644be6fb P 48c41a1044904ed1adc033d619ddcc8b [LEADER]: Connected to new peer: Peer: permanent_uuid: "be1dee3ca50746e697d6e3b29d65d3fa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:21.898216 5491 mvcc.cc:205] Tried to move back new op lower bound from 7333264432719740928 to 7333264432626327552. Current Snapshot: MvccSnapshot[applied={T|T < 7333264432719740928}]
I20260925 14:49:21.898866 5492 mvcc.cc:205] Tried to move back new op lower bound from 7333264432719740928 to 7333264432626327552. Current Snapshot: MvccSnapshot[applied={T|T < 7333264432719740928}]
I20260925 14:49:21.901311 5013 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:21.901409 5013 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:21.901589 5493 mvcc.cc:205] Tried to move back new op lower bound from 7333264432719740928 to 7333264432626327552. Current Snapshot: MvccSnapshot[applied={T|T < 7333264432719740928}]
I20260925 14:49:21.902143 5013 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cdce463646da43f5b0040c0727d545be: Sending DeleteTablet for 3 replicas of tablet c5e74ada7ad1490994749656644be6fb
I20260925 14:49:21.902462 5136 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e74ada7ad1490994749656644be6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='slave'} at 127.0.0.1:46102
I20260925 14:49:21.902525 5267 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e74ada7ad1490994749656644be6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='slave'} at 127.0.0.1:42550
I20260925 14:49:21.902644 5135 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e74ada7ad1490994749656644be6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='slave'} at 127.0.0.1:46102
I20260925 14:49:21.902699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5074
I20260925 14:49:21.902799 5398 tablet_service.cc:1514] Processing DeleteTablet for tablet c5e74ada7ad1490994749656644be6fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:21 UTC) from {username='slave'} at 127.0.0.1:36540
I20260925 14:49:21.903239 5503 tablet_replica.cc:330] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: stopping tablet replica
I20260925 14:49:21.903333 5503 raft_consensus.cc:2229] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:21.903410 5503 raft_consensus.cc:2258] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:21.903735 5502 tablet_replica.cc:330] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: stopping tablet replica
I20260925 14:49:21.903848 5503 ts_tablet_manager.cc:1905] T c5e74ada7ad1490994749656644be6fb P 157a322da53b4215832644e2284319b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:21.903857 5502 raft_consensus.cc:2229] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:21.903959 5502 raft_consensus.cc:2258] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:21.904341 5502 ts_tablet_manager.cc:1905] T c5e74ada7ad1490994749656644be6fb P be1dee3ca50746e697d6e3b29d65d3fa: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:49:21.909447 5000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42787 (error 108)
I20260925 14:49:21.909940 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.909812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42787 (local address 127.0.0.1:46126)
0925 14:49:21.909870 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.909911 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42787: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:49:21.909945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5205
W20260925 14:49:21.910065 5000 catalog_manager.cc:4584] TS 48c41a1044904ed1adc033d619ddcc8b (127.0.98.1:42787): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c5e74ada7ad1490994749656644be6fb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42787: connect: Connection refused (error 111)
I20260925 14:49:21.929764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5336
I20260925 14:49:21.929857 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.929717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40195 (local address 127.0.0.1:42560)
0925 14:49:21.929780 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.929818 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:49:21.934911 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 4983
I20260925 14:49:21.934955 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:21.934830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41885 (local address 127.0.0.1:36554)
0925 14:49:21.934886 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:21.934921 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:41885: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
2026-09-25T14:49:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (675 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:49:21.954913 392 mini_oidc.cc:152] Starting JWKS server
I20260925 14:49:21.955957 392 webserver.cc:460] Webserver started at https://127.0.0.1:44845/ using document root <none> and password file <none>
I20260925 14:49:21.956060 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:49:21.956209 392 webserver.cc:460] Webserver started at http://127.0.0.1:40035/ using document root <none> and password file <none>
2026-09-25T14:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:21Z Disabled control of system clock
I20260925 14:49:21.963805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41399
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41399
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/testchainca.cert with env {}
W20260925 14:49:22.047112 5510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:22.047374 5510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:22.047411 5510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:22.048620 5510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:22.048694 5510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:22.048717 5510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:22.048774 5510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:22.048796 5510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:49:22.048822 5510 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:49:22.048851 5510 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44845/jwks.json
I20260925 14:49:22.050506 5510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:22.050791 5510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:22.051019 5510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:22.056373 5510 server_base.cc:1034] running on GCE node
W20260925 14:49:22.056257 5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.056476 5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.056272 5518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:22.056803 5510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:22.057120 5510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:22.058287 5510 hybrid_clock.cc:648] HybridClock initialized: now 1790347762058273 us; error 43 us; skew 500 ppm
I20260925 14:49:22.059633 5510 webserver.cc:460] Webserver started at http://127.0.98.62:44361/ using document root <none> and password file <none>
I20260925 14:49:22.059916 5510 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:22.059988 5510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:22.060140 5510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:22.061251 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "64574aec022f494391a71f65f2119396"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "d20e3c81b39f05fc1179a5fc1aad6c50"
server_key_iv: "9829efcf37848124660f066b575c7de4"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.061609 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "64574aec022f494391a71f65f2119396"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "d20e3c81b39f05fc1179a5fc1aad6c50"
server_key_iv: "9829efcf37848124660f066b575c7de4"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.063037 5510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:22.063767 5524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:22.063934 5510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:22.064031 5510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "64574aec022f494391a71f65f2119396"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "d20e3c81b39f05fc1179a5fc1aad6c50"
server_key_iv: "9829efcf37848124660f066b575c7de4"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.064114 5510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:22.084722 5510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:22.085083 5510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:22.085242 5510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:22.089468 5510 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41399
I20260925 14:49:22.089535 5576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41399 every 8 connection(s)
I20260925 14:49:22.089916 5510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:22.090979 5577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:22.093403 5577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396: Bootstrap starting.
I20260925 14:49:22.093981 5577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:22.094274 5577 log.cc:824] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:22.095201 5577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396: No bootstrap required, opened a new log
I20260925 14:49:22.096786 5577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64574aec022f494391a71f65f2119396" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41399 } }
I20260925 14:49:22.096933 5577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:22.096962 5577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64574aec022f494391a71f65f2119396, State: Initialized, Role: FOLLOWER
I20260925 14:49:22.097124 5577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64574aec022f494391a71f65f2119396" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41399 } }
I20260925 14:49:22.097210 5577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:22.097234 5577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:22.097280 5577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:22.098482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5510
I20260925 14:49:22.098567 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:22.098708 392 external_mini_cluster.cc:1428] Setting key f82416ab99b52fd63b538fd63087467a
I20260925 14:49:22.104061 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.099212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.0.98.62:41399)
0925 14:49:22.099483 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.099491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.099517 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:49:22.099611 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.099617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.099674 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.099837 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:22.100045 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.101238 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.102059 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.102540 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.102671 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.103408 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:22.103413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:22.103413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:22.103583 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:22.103615 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:22.103627 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:22.103700 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:22.103784 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:49:22.103892 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":105,"threads_started":1}
I20260925 14:49:22.105173 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:49:22.106829 5577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64574aec022f494391a71f65f2119396" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41399 } }
I20260925 14:49:22.106971 5577 leader_election.cc:302] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64574aec022f494391a71f65f2119396; no voters:
I20260925 14:49:22.107155 5577 leader_election.cc:288] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:22.107218 5582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:22.107419 5582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [term 1 LEADER]: Becoming Leader. State: Replica: 64574aec022f494391a71f65f2119396, State: Running, Role: LEADER
I20260925 14:49:22.107425 5577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:22.107537 5582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64574aec022f494391a71f65f2119396" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41399 } }
I20260925 14:49:22.107858 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64574aec022f494391a71f65f2119396. Latest consensus state: current_term: 1 leader_uuid: "64574aec022f494391a71f65f2119396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64574aec022f494391a71f65f2119396" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41399 } } }
I20260925 14:49:22.107954 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.107868 5584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64574aec022f494391a71f65f2119396" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64574aec022f494391a71f65f2119396" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41399 } } }
I20260925 14:49:22.108089 5584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396 [sys.catalog]: This master's current role is: LEADER
W20260925 14:49:22.109457 5597 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:49:22.110718 5597 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:49:22.110797 5587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:22.110942 5587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:22.112426 5587 catalog_manager.cc:1347] Generated new cluster ID: 4a2e592ba51c4fe5a3844c4e5da52960
I20260925 14:49:22.112476 5587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:22.123286 5587 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:22.124040 5587 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:22.128988 5587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64574aec022f494391a71f65f2119396: Generated new TSK 0
I20260925 14:49:22.129200 5587 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:49:22.156927 5607 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:49:22.157048 5607 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:49:22.157325 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.140874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.0.98.62:41399)
0925 14:49:22.140941 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.140944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.141114 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:49:22.141211 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.141212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.141222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.141282 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:49:22.141447 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.141946 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.143113 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.143297 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.143352 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.143575 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:49:22.157080 (+ 13505us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:49:22.157260 (+ 180us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56640: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:49:22.157398 5581 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56640: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 14:49:22.157791 5606 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:22.140741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41399 (local address 127.0.0.1:56640)
0925 14:49:22.141024 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:22.141036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:22.141117 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:22.141305 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:22.141306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:22.141308 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:49:22.141381 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.141383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.142034 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.142035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.143024 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.143027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.143372 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.143376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.143444 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.143466 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:49:22.157281 (+ 13815us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:49:22.157430 (+ 149us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:41399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 14:49:22.158335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5510
2026-09-25T14:49:22Z chronyd exiting
I20260925 14:49:22.195334 392 test_util.cc:175] -----------------------------------------------
I20260925 14:49:22.195415 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790347739045519-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:49:22.196449 392 mini_oidc.cc:152] Starting JWKS server
I20260925 14:49:22.197553 392 webserver.cc:460] Webserver started at https://127.0.0.1:33403/ using document root <none> and password file <none>
I20260925 14:49:22.197613 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:49:22.197763 392 webserver.cc:460] Webserver started at http://127.0.0.1:34143/ using document root <none> and password file <none>
2026-09-25T14:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:22Z Disabled control of system clock
I20260925 14:49:22.207623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44363
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44363
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33403/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/testchainca.cert with env {}
W20260925 14:49:22.288276 5613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:22.288491 5613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:22.288512 5613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:22.289641 5613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:22.289685 5613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:22.289697 5613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:22.289710 5613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:22.289722 5613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:49:22.289734 5613 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:49:22.289745 5613 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33403/jwks.json
I20260925 14:49:22.291375 5613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44363
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44363
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33403/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:22.291587 5613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:22.291746 5613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:22.296628 5619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.296634 5621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.296618 5618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:22.296770 5613 server_base.cc:1034] running on GCE node
I20260925 14:49:22.297098 5613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:22.297425 5613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:22.298592 5613 hybrid_clock.cc:648] HybridClock initialized: now 1790347762298566 us; error 44 us; skew 500 ppm
I20260925 14:49:22.299889 5613 webserver.cc:460] Webserver started at http://127.0.98.62:37161/ using document root <none> and password file <none>
I20260925 14:49:22.300127 5613 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:22.300179 5613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:22.300372 5613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:22.301944 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "bf3720c6ed11421cb370af9384ea7b02"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "227805d60383dec323a74c754d6d8d16"
server_key_iv: "72b30070e8f7a9f618e4e03ff514c915"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.302315 5613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf3720c6ed11421cb370af9384ea7b02"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "227805d60383dec323a74c754d6d8d16"
server_key_iv: "72b30070e8f7a9f618e4e03ff514c915"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.303566 5613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:22.304378 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:22.304558 5613 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:22.304651 5613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "bf3720c6ed11421cb370af9384ea7b02"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "227805d60383dec323a74c754d6d8d16"
server_key_iv: "72b30070e8f7a9f618e4e03ff514c915"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.304723 5613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:22.320591 5613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:22.320917 5613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:22.321066 5613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:22.325098 5613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44363
I20260925 14:49:22.325125 5679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44363 every 8 connection(s)
I20260925 14:49:22.325490 5613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:22.326489 5680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:22.328677 5680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02: Bootstrap starting.
I20260925 14:49:22.329197 5680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:22.329424 5680 log.cc:824] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:22.330042 5680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02: No bootstrap required, opened a new log
I20260925 14:49:22.331648 5680 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3720c6ed11421cb370af9384ea7b02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260925 14:49:22.331813 5680 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:22.331875 5680 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf3720c6ed11421cb370af9384ea7b02, State: Initialized, Role: FOLLOWER
I20260925 14:49:22.332005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5613
I20260925 14:49:22.332033 5680 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3720c6ed11421cb370af9384ea7b02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260925 14:49:22.332087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:22.332123 5680 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:22.332165 5680 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:22.332218 5680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:22.332280 392 external_mini_cluster.cc:1428] Setting key 08522ffc29a9f4e9098d665f6747a73c
I20260925 14:49:22.332965 5680 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3720c6ed11421cb370af9384ea7b02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260925 14:49:22.333074 5680 leader_election.cc:302] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf3720c6ed11421cb370af9384ea7b02; no voters:
I20260925 14:49:22.333214 5680 leader_election.cc:288] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:22.333241 5685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:22.333395 5685 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [term 1 LEADER]: Becoming Leader. State: Replica: bf3720c6ed11421cb370af9384ea7b02, State: Running, Role: LEADER
I20260925 14:49:22.333457 5680 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:22.333493 5685 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3720c6ed11421cb370af9384ea7b02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } }
I20260925 14:49:22.333797 5686 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf3720c6ed11421cb370af9384ea7b02. Latest consensus state: current_term: 1 leader_uuid: "bf3720c6ed11421cb370af9384ea7b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3720c6ed11421cb370af9384ea7b02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } } }
I20260925 14:49:22.333899 5687 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf3720c6ed11421cb370af9384ea7b02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf3720c6ed11421cb370af9384ea7b02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44363 } } }
I20260925 14:49:22.333904 5686 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.333987 5687 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.334563 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:22.335016 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:22.336845 5693 catalog_manager.cc:1347] Generated new cluster ID: b1e4e5affc0c43a5b8e755a9d733b993
I20260925 14:49:22.336901 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:22.339811 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.332756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.0.98.62:44363)
0925 14:49:22.332996 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.333005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.333040 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:49:22.333158 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.333166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.333239 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.333400 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:22.333816 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.335623 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.336815 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.337734 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.338258 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.338732 (+ 474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:22.338736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:22.338736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:22.339152 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:22.339191 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:22.339205 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:22.339487 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:22.339519 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 14:49:22.339597 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260925 14:49:22.340776 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:49:22.350968 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:22.351542 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:22.354825 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf3720c6ed11421cb370af9384ea7b02: Generated new TSK 0
I20260925 14:49:22.355011 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:49:22.386333 5710 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:49:22.386523 5710 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 14:49:22.386900 5709 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:22.363877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44363 (local address 127.0.0.1:34140)
0925 14:49:22.364049 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:22.364062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:22.364135 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:22.364314 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:22.364315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:22.364317 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:49:22.364380 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.364383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.365028 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.365029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.365989 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.365992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.366553 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.366557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.366634 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.366641 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:49:22.386702 (+ 20061us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:49:22.386802 (+ 100us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44363: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260925 14:49:22.387216 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.363947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.0.98.62:44363)
0925 14:49:22.364012 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.364014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.364132 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:49:22.364201 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.364202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.364215 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.364289 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:49:22.364445 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.364942 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.366123 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.366460 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.366569 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.366757 (+ 188us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:49:22.386537 (+ 19780us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:49:22.387151 (+ 614us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 14:49:22.387272 5684 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260925 14:49:22.387784 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5613
2026-09-25T14:49:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:49:22.426575 392 mini_oidc.cc:152] Starting JWKS server
I20260925 14:49:22.427237 392 webserver.cc:460] Webserver started at https://127.0.0.1:44071/ using document root <none> and password file <none>
I20260925 14:49:22.427315 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:49:22.427464 392 webserver.cc:460] Webserver started at http://127.0.0.1:36807/ using document root <none> and password file <none>
2026-09-25T14:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:22Z Disabled control of system clock
I20260925 14:49:22.435194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43043
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43043
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44071/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:49:22.515573 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:22.515784 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:22.515805 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:22.516870 5716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:22.516903 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:22.516916 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:22.516928 5716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:22.516940 5716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:49:22.516952 5716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:49:22.516963 5716 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44071/jwks.json
I20260925 14:49:22.518210 5716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44071/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:22.518458 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:22.518750 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:22.523437 5721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.523455 5724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.523455 5722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:22.523612 5716 server_base.cc:1034] running on GCE node
I20260925 14:49:22.523882 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:22.524150 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:22.525339 5716 hybrid_clock.cc:648] HybridClock initialized: now 1790347762525316 us; error 36 us; skew 500 ppm
I20260925 14:49:22.526718 5716 webserver.cc:460] Webserver started at http://127.0.98.62:39613/ using document root <none> and password file <none>
I20260925 14:49:22.526973 5716 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:22.527045 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:22.527194 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:22.528251 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "8d4c388b0cfd42a9aca32f244bca1a27"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "25235c3047faed727928c185059ebe56"
server_key_iv: "1b885187d9adabdfe9774cab57d829de"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.528630 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d4c388b0cfd42a9aca32f244bca1a27"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "25235c3047faed727928c185059ebe56"
server_key_iv: "1b885187d9adabdfe9774cab57d829de"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.529987 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:22.530753 5730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:22.530925 5716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:22.531049 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "8d4c388b0cfd42a9aca32f244bca1a27"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "25235c3047faed727928c185059ebe56"
server_key_iv: "1b885187d9adabdfe9774cab57d829de"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.531119 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:22.560461 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:22.560801 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:22.560967 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:22.565064 5716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43043
I20260925 14:49:22.565101 5782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43043 every 8 connection(s)
I20260925 14:49:22.565454 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:22.566140 5783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:22.568727 5783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27: Bootstrap starting.
I20260925 14:49:22.569310 5783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:22.569555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5716
I20260925 14:49:22.569625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:22.569614 5783 log.cc:824] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:22.569820 392 external_mini_cluster.cc:1428] Setting key 0f09761a6dd0c7585302ebaf2fb4947c
I20260925 14:49:22.570372 5783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27: No bootstrap required, opened a new log
I20260925 14:49:22.571977 5783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260925 14:49:22.572134 5783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:22.572171 5783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d4c388b0cfd42a9aca32f244bca1a27, State: Initialized, Role: FOLLOWER
I20260925 14:49:22.572312 5783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260925 14:49:22.572402 5783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:22.572451 5783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:22.572499 5783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:22.573199 5783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260925 14:49:22.573282 5783 leader_election.cc:302] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d4c388b0cfd42a9aca32f244bca1a27; no voters:
I20260925 14:49:22.573449 5783 leader_election.cc:288] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:22.573477 5788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:22.573576 5788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [term 1 LEADER]: Becoming Leader. State: Replica: 8d4c388b0cfd42a9aca32f244bca1a27, State: Running, Role: LEADER
I20260925 14:49:22.573657 5788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } }
I20260925 14:49:22.573699 5783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:22.573969 5790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d4c388b0cfd42a9aca32f244bca1a27. Latest consensus state: current_term: 1 leader_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } } }
I20260925 14:49:22.574056 5790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.574208 5789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c388b0cfd42a9aca32f244bca1a27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43043 } } }
I20260925 14:49:22.574258 5789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.574364 5793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:22.574870 5793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:22.576757 5793 catalog_manager.cc:1347] Generated new cluster ID: 29d1c31816bd46e4940157577b43f146
I20260925 14:49:22.576853 5793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:22.577116 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.570331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46084 (local address 127.0.98.62:43043)
0925 14:49:22.570573 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.570580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.570604 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:49:22.571962 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.571968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.572024 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.572157 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:22.572356 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.573407 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.574672 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.575330 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.575455 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.576401 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:22.576404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:22.576405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:22.576606 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:22.576648 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:22.576663 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:22.576735 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:22.576837 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:49:22.576916 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":120,"threads_started":1}
I20260925 14:49:22.578053 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:49:22.593397 5793 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:22.593911 5793 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:22.600380 5793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d4c388b0cfd42a9aca32f244bca1a27: Generated new TSK 0
I20260925 14:49:22.600560 5793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:22.612403 5787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.608793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.98.62:43043)
0925 14:49:22.608857 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.608859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.608942 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:49:22.608983 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.608984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.608995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.609077 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:49:22.609189 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.609967 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.610764 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.611068 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.611141 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.611281 (+ 140us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:49:22.612196 (+ 915us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:49:22.612325 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 14:49:22.612455 5787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46094: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 14:49:22.612479 5812 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:22.608610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43043 (local address 127.0.0.1:46094)
0925 14:49:22.608866 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:22.608884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:22.608937 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:22.609079 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:22.609080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:22.609082 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:49:22.609127 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.609129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.610039 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.610039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.610689 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.610691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.611143 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.611148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.611187 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.611191 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:49:22.612296 (+ 1105us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:49:22.612385 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:43043: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44071/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":210,"threads_started":1}
I20260925 14:49:22.612999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5716
E20260925 14:49:22.624957 5813 webserver.cc:533] Webserver: sslize failed:
E20260925 14:49:22.625034 5813 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T14:49:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:49:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:22Z Disabled control of system clock
I20260925 14:49:22.658473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36489
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/testchainca.cert with env {}
W20260925 14:49:22.739885 5817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:22.740094 5817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:22.740116 5817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:22.741206 5817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:22.741240 5817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:22.741253 5817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:22.741266 5817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:22.741277 5817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:49:22.741323 5817 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:49:22.741336 5817 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:49:22.742825 5817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:22.743018 5817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:22.743173 5817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:22.748034 5822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.748036 5825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:22.748086 5823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:22.748208 5817 server_base.cc:1034] running on GCE node
I20260925 14:49:22.748461 5817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:22.748754 5817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:22.749917 5817 hybrid_clock.cc:648] HybridClock initialized: now 1790347762749904 us; error 46 us; skew 500 ppm
I20260925 14:49:22.751318 5817 webserver.cc:460] Webserver started at http://127.0.98.62:45211/ using document root <none> and password file <none>
I20260925 14:49:22.751574 5817 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:22.751647 5817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:22.751782 5817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:22.752830 5817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "ae4fcda60b154df287708c49b81cf1e8"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "4f144c3675ee21be1e77553fd6131720"
server_key_iv: "ff3e1271d5d5afb4ed5fc33af9f2f72d"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.753216 5817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae4fcda60b154df287708c49b81cf1e8"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "4f144c3675ee21be1e77553fd6131720"
server_key_iv: "ff3e1271d5d5afb4ed5fc33af9f2f72d"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.754555 5817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:22.755290 5831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:22.755455 5817 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:22.755560 5817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "ae4fcda60b154df287708c49b81cf1e8"
format_stamp: "Formatted at 2026-09-25 14:49:22 on dist-test-slave-z2s5"
server_key: "4f144c3675ee21be1e77553fd6131720"
server_key_iv: "ff3e1271d5d5afb4ed5fc33af9f2f72d"
server_key_version: "encryptionkey@0"
I20260925 14:49:22.755635 5817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:22.766614 5817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:22.766928 5817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:22.767107 5817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:22.771039 5817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36489
I20260925 14:49:22.771070 5883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36489 every 8 connection(s)
I20260925 14:49:22.771423 5817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:22.772418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5817
I20260925 14:49:22.772459 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:22.772529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:22.772727 392 external_mini_cluster.cc:1428] Setting key 653e661c5fc40b94345d7f15fc393d0a
I20260925 14:49:22.775556 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8: Bootstrap starting.
I20260925 14:49:22.776250 5884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:22.776539 5884 log.cc:824] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:22.777200 5884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8: No bootstrap required, opened a new log
I20260925 14:49:22.778349 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.773253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33610 (local address 127.0.98.62:36489)
0925 14:49:22.773465 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.773482 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.773507 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:49:22.773947 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.773955 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.774022 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.774136 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:22.774433 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.775321 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.776417 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.776859 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.776996 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.777750 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:22.777754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:22.777754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:22.777922 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:22.777953 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:22.777964 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:22.778035 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:22.778110 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:49:22.778188 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":108,"threads_started":1}
I20260925 14:49:22.779448 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:49:22.779358 5884 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4fcda60b154df287708c49b81cf1e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260925 14:49:22.779542 5884 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:22.779587 5884 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae4fcda60b154df287708c49b81cf1e8, State: Initialized, Role: FOLLOWER
I20260925 14:49:22.779709 5884 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4fcda60b154df287708c49b81cf1e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260925 14:49:22.779788 5884 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:22.779837 5884 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:22.779886 5884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:22.780550 5884 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4fcda60b154df287708c49b81cf1e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260925 14:49:22.780658 5884 leader_election.cc:302] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae4fcda60b154df287708c49b81cf1e8; no voters:
I20260925 14:49:22.780813 5884 leader_election.cc:288] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:22.780869 5889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:22.781019 5889 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [term 1 LEADER]: Becoming Leader. State: Replica: ae4fcda60b154df287708c49b81cf1e8, State: Running, Role: LEADER
I20260925 14:49:22.781049 5884 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:22.781106 5889 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4fcda60b154df287708c49b81cf1e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } }
I20260925 14:49:22.781481 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae4fcda60b154df287708c49b81cf1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4fcda60b154df287708c49b81cf1e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } } }
I20260925 14:49:22.781481 5891 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae4fcda60b154df287708c49b81cf1e8. Latest consensus state: current_term: 1 leader_uuid: "ae4fcda60b154df287708c49b81cf1e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae4fcda60b154df287708c49b81cf1e8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36489 } } }
I20260925 14:49:22.781594 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.781600 5891 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:22.781814 5897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:22.782281 5897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:22.783834 5897 catalog_manager.cc:1347] Generated new cluster ID: c6b091b3b8cb4361ad8088e5be54aa13
I20260925 14:49:22.783908 5897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:22.814420 5897 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:22.815014 5897 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:22.819990 5897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae4fcda60b154df287708c49b81cf1e8: Generated new TSK 0
I20260925 14:49:22.820199 5897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:22.952629 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:22.834736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.98.62:36489)
0925 14:49:22.834808 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:22.834810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:22.834923 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:49:22.835004 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:22.835005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:22.835018 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:22.835089 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:49:22.835245 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.835779 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.836587 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.836802 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.836895 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.837093 (+ 198us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:49:22.952338 (+115245us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:49:22.952535 (+ 197us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33614: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 14:49:22.952720 5887 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33614: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 14:49:22.952721 5913 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:22.834675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36489 (local address 127.0.0.1:33614)
0925 14:49:22.834853 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:22.834869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:22.834941 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:22.835094 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:22.835095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:22.835097 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:49:22.835166 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.835169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.835888 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.835889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.836497 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:22.836499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:22.836904 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:22.836911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:22.836969 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:22.836979 (+ 10us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:49:22.952492 (+115513us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:49:22.952609 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36489: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":101,"threads_started":1}
I20260925 14:49:22.953118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5817
2026-09-25T14:49:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (319 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (86 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:23 dist-test-slave-z2s5 krb5kdc[5925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:23 dist-test-slave-z2s5 krb5kdc[5925](info): set up 2 sockets
Sep 25 14:49:23 dist-test-slave-z2s5 krb5kdc[5925](info): commencing operation
krb5kdc: starting...
W20260925 14:49:24.154657 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:24.273860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38977
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38977
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:24.352676 5938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:24.352932 5938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:24.352969 5938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:24.354115 5938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:24.354193 5938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:24.354218 5938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:24.354244 5938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:24.354274 5938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:24.355840 5938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:24.356119 5938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:24.356340 5938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:24.361366 5943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.361403 5938 server_base.cc:1034] running on GCE node
W20260925 14:49:24.361392 5944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:24.361421 5946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.361807 5938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:24.362108 5938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:24.363272 5938 hybrid_clock.cc:648] HybridClock initialized: now 1790347764363250 us; error 53 us; skew 500 ppm
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.372140 5938 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:24.372581 5938 webserver.cc:460] Webserver started at http://127.0.98.62:34975/ using document root <none> and password file <none>
I20260925 14:49:24.372764 5938 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:24.372804 5938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:24.372896 5938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:24.373951 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "e9b913468eeb43d1812cb0c932f1ee59"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "c5c00ddbb269cb1e21655bcb4614f95a"
server_key_iv: "37b3c7e2c33bf28fdc22c45e96f69700"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.374259 5938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9b913468eeb43d1812cb0c932f1ee59"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "c5c00ddbb269cb1e21655bcb4614f95a"
server_key_iv: "37b3c7e2c33bf28fdc22c45e96f69700"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.375640 5938 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:24.376439 5953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.376597 5938 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:24.376688 5938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "e9b913468eeb43d1812cb0c932f1ee59"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "c5c00ddbb269cb1e21655bcb4614f95a"
server_key_iv: "37b3c7e2c33bf28fdc22c45e96f69700"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.376746 5938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:24.392717 5938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:24.395515 5938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:24.395677 5938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:24.400065 5938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38977
I20260925 14:49:24.400151 6005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38977 every 8 connection(s)
I20260925 14:49:24.400667 5938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:24.401376 6006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:24.403740 6006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59: Bootstrap starting.
I20260925 14:49:24.404357 6006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:24.404646 6006 log.cc:824] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:24.405299 6006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59: No bootstrap required, opened a new log
I20260925 14:49:24.406920 6006 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b913468eeb43d1812cb0c932f1ee59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38977 } }
I20260925 14:49:24.407055 6006 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:24.407078 6006 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b913468eeb43d1812cb0c932f1ee59, State: Initialized, Role: FOLLOWER
I20260925 14:49:24.407269 6006 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b913468eeb43d1812cb0c932f1ee59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38977 } }
I20260925 14:49:24.407363 6006 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:24.407405 6006 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:24.407454 6006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:24.408336 6006 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b913468eeb43d1812cb0c932f1ee59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38977 } }
I20260925 14:49:24.408447 6006 leader_election.cc:302] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b913468eeb43d1812cb0c932f1ee59; no voters:
I20260925 14:49:24.408640 6006 leader_election.cc:288] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:24.408722 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:24.408846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 5938
I20260925 14:49:24.408917 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [term 1 LEADER]: Becoming Leader. State: Replica: e9b913468eeb43d1812cb0c932f1ee59, State: Running, Role: LEADER
I20260925 14:49:24.408957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:24.408926 6006 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:24.409017 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b913468eeb43d1812cb0c932f1ee59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38977 } }
I20260925 14:49:24.409173 392 external_mini_cluster.cc:1428] Setting key efea27f19843e1340b4f71e16c3ed370
I20260925 14:49:24.409322 6010 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b913468eeb43d1812cb0c932f1ee59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b913468eeb43d1812cb0c932f1ee59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38977 } } }
I20260925 14:49:24.409425 6010 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:24.409312 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b913468eeb43d1812cb0c932f1ee59. Latest consensus state: current_term: 1 leader_uuid: "e9b913468eeb43d1812cb0c932f1ee59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b913468eeb43d1812cb0c932f1ee59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38977 } } }
I20260925 14:49:24.409603 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:24.409721 6015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:24.410904 6015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:24.412601 6015 catalog_manager.cc:1347] Generated new cluster ID: 4011be7b799742aebedfe3e9ac3bb69c
I20260925 14:49:24.412657 6015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:24.420809 6015 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:24.421176 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.409795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.0.98.62:38977)
0925 14:49:24.409967 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.409978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.410201 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:49:24.410357 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.410365 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.410448 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.410594 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.411455 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.412708 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.413730 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.414319 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.414426 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.417283 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.417289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.417290 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.417306 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.419363 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.419770 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.419773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.419774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.419884 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.420092 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.420092 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.420093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.420392 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.420547 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.420802 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.420954 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260925 14:49:24.421489 6015 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:24.423990 6015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b913468eeb43d1812cb0c932f1ee59: Generated new TSK 0
I20260925 14:49:24.424161 6015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:24.454152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38977
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:24.533960 6032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:24.534178 6032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:24.534204 6032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:24.535481 6032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:24.535533 6032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:24.535561 6032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:24.537135 6032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:24.537415 6032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:24.537654 6032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:24.542717 6038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:24.542717 6037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:24.542732 6040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.542933 6032 server_base.cc:1034] running on GCE node
I20260925 14:49:24.543184 6032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:24.543457 6032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:24.544620 6032 hybrid_clock.cc:648] HybridClock initialized: now 1790347764544609 us; error 46 us; skew 500 ppm
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.547086 6032 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:24.547547 6032 webserver.cc:460] Webserver started at http://127.0.98.1:34849/ using document root <none> and password file <none>
I20260925 14:49:24.547789 6032 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:24.547861 6032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:24.548017 6032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:24.549103 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "6df703198af74d80b3194a368a7a913d"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "c282f8e7fa40b32605290d467f38f85c"
server_key_iv: "46942b542bc45276771a3b781c54a761"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.549468 6032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6df703198af74d80b3194a368a7a913d"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "c282f8e7fa40b32605290d467f38f85c"
server_key_iv: "46942b542bc45276771a3b781c54a761"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.550864 6032 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:24.551605 6047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.551779 6032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:24.551880 6032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "6df703198af74d80b3194a368a7a913d"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "c282f8e7fa40b32605290d467f38f85c"
server_key_iv: "46942b542bc45276771a3b781c54a761"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.551961 6032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:24.570765 6032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:24.572832 6032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:24.572997 6032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:24.573344 6032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:24.573683 6032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:24.573740 6032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.573772 6032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:24.573803 6032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.579804 6032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43001
I20260925 14:49:24.579879 6160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43001 every 8 connection(s)
I20260925 14:49:24.580212 6032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:24.589123 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.581006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44863 (local address 127.0.98.62:38977)
0925 14:49:24.581055 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.581057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.581277 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:49:24.582202 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.582203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.582214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.582277 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.582958 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.583475 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.584433 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.584630 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.584680 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.586798 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.586799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.586799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.586806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.588016 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.588365 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.588366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.588367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.588413 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.588624 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.588625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.588625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.588720 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.588768 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.588992 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.589058 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:24.589313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6032
I20260925 14:49:24.589301 6163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.580968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38977 (local address 127.0.98.1:44863)
0925 14:49:24.581144 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:24.581187 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:24.582108 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:24.582341 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:24.582345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:24.582678 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:24.582878 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:24.582883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.583555 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.583555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:24.584359 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:24.584361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.584705 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.584705 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:24.584808 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.585369 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:24.585377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:24.586715 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:24.588086 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:24.588091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:24.588094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:24.588301 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:24.588470 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:24.588471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:24.588471 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:24.588572 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:24.588772 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:24.588776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:24.588883 (+ 107us) client_negotiation.cc:769] Sending connection context
0925 14:49:24.588963 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:49:24.589076 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260925 14:49:24.589424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:24.589609 392 external_mini_cluster.cc:1428] Setting key e8a8d2cdd06a990c2f03276c5512d276
I20260925 14:49:24.589933 6161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38977
I20260925 14:49:24.590056 6161 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:24.590265 6161 heartbeater.cc:511] Master 127.0.98.62:38977 requested a full tablet report, sending...
I20260925 14:49:24.590853 5970 ts_manager.cc:194] Registered new tserver with Master: 6df703198af74d80b3194a368a7a913d (127.0.98.1:43001)
I20260925 14:49:24.591758 5970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44863
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:24.615754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38977
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:24.697129 6166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:24.697329 6166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:24.697352 6166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:24.698563 6166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:24.698606 6166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:24.698678 6166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:24.700176 6166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:24.700430 6166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:24.700672 6166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:24.705602 6174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.705752 6166 server_base.cc:1034] running on GCE node
W20260925 14:49:24.705574 6172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:24.705585 6171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.706121 6166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:24.706329 6166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:24.707468 6166 hybrid_clock.cc:648] HybridClock initialized: now 1790347764707456 us; error 28 us; skew 500 ppm
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.710098 6166 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:24.710687 6166 webserver.cc:460] Webserver started at http://127.0.98.2:44815/ using document root <none> and password file <none>
I20260925 14:49:24.710943 6166 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:24.710994 6166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:24.711136 6166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:24.712170 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "7af952f97bf146ff9057300afe6f8d27"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "f107b976d38becbed3ce3f9c951fff6c"
server_key_iv: "f4c922cd6e48ff0d53bc4b26c6607577"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.712525 6166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7af952f97bf146ff9057300afe6f8d27"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "f107b976d38becbed3ce3f9c951fff6c"
server_key_iv: "f4c922cd6e48ff0d53bc4b26c6607577"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.713791 6166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:24.714490 6181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.714650 6166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:24.714735 6166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "7af952f97bf146ff9057300afe6f8d27"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "f107b976d38becbed3ce3f9c951fff6c"
server_key_iv: "f4c922cd6e48ff0d53bc4b26c6607577"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.714807 6166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:24.724915 6166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:24.726879 6166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:24.727042 6166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:24.727279 6166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:24.727595 6166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:24.727631 6166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.727653 6166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:24.727666 6166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.733570 6166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40995
I20260925 14:49:24.733618 6294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40995 every 8 connection(s)
I20260925 14:49:24.733944 6166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:24.740635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6166
I20260925 14:49:24.740744 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:24.740962 392 external_mini_cluster.cc:1428] Setting key db2d935cf9a1c694f9e415b6bf35d546
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:24.744962 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.734697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53135 (local address 127.0.98.62:38977)
0925 14:49:24.734787 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.734790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.734964 (+ 174us) server_negotiation.cc:407] Connection header received
0925 14:49:24.735883 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.735884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.735894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.735949 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.736542 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.737278 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.738776 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.739102 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.739181 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.741748 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.741750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.741750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.741758 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.743384 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.743876 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.743876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.743877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.743937 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.744217 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.744218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.744218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.744376 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.744444 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.744699 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.744801 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:49:24.745024 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.734659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38977 (local address 127.0.98.2:53135)
0925 14:49:24.734828 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:24.734870 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:24.735786 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:24.736012 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:24.736016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:24.736268 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:24.736470 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:24.736474 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.737374 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.737376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:24.738680 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:24.738684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.739204 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.739206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:24.739350 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.740031 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:24.740042 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:24.741665 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:24.743465 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:24.743472 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:24.743474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:24.743797 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:24.743981 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:24.743982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:24.743983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:24.744125 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:24.744441 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:24.744446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:24.744565 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:49:24.744654 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:49:24.744783 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260925 14:49:24.745570 6295 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38977
I20260925 14:49:24.745683 6295 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:24.745869 6295 heartbeater.cc:511] Master 127.0.98.62:38977 requested a full tablet report, sending...
I20260925 14:49:24.746418 5970 ts_manager.cc:194] Registered new tserver with Master: 7af952f97bf146ff9057300afe6f8d27 (127.0.98.2:40995)
I20260925 14:49:24.746871 5970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53135
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:24.767571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38977
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:24.846369 6300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:24.846634 6300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:24.846673 6300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:24.847853 6300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:24.847926 6300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:24.847971 6300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:24.849438 6300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:24.849713 6300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:24.849951 6300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:24.855042 6308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.855124 6300 server_base.cc:1034] running on GCE node
W20260925 14:49:24.855019 6306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:24.855019 6305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:24.855520 6300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:24.855728 6300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:24.856860 6300 hybrid_clock.cc:648] HybridClock initialized: now 1790347764856857 us; error 33 us; skew 500 ppm
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.859355 6300 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:24.859797 6300 webserver.cc:460] Webserver started at http://127.0.98.3:34627/ using document root <none> and password file <none>
I20260925 14:49:24.860021 6300 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:24.860088 6300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:24.860236 6300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:24.861281 6300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "e4bf9862dc614fb2b73a7fae95ab2512"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "931a5131441a7a4e692e516a995d93b8"
server_key_iv: "d7bdc2b9534ccce0b7153aae0d0aae82"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.861682 6300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e4bf9862dc614fb2b73a7fae95ab2512"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "931a5131441a7a4e692e516a995d93b8"
server_key_iv: "d7bdc2b9534ccce0b7153aae0d0aae82"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.863089 6300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:24.863811 6315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.864032 6300 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:24.864151 6300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "e4bf9862dc614fb2b73a7fae95ab2512"
format_stamp: "Formatted at 2026-09-25 14:49:24 on dist-test-slave-z2s5"
server_key: "931a5131441a7a4e692e516a995d93b8"
server_key_iv: "d7bdc2b9534ccce0b7153aae0d0aae82"
server_key_version: "encryptionkey@0"
I20260925 14:49:24.864274 6300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:24.875495 6300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:24.877688 6300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:24.877847 6300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:24.878082 6300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:24.878440 6300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:24.878499 6300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.878542 6300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:24.878568 6300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:24.884341 6300 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39433
I20260925 14:49:24.884383 6428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39433 every 8 connection(s)
I20260925 14:49:24.884696 6300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:24.892123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6300
I20260925 14:49:24.892207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:24.892419 392 external_mini_cluster.cc:1428] Setting key b9307b1b6e30506443047b40b377b992
I20260925 14:49:24.894250 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.885394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58131 (local address 127.0.98.62:38977)
0925 14:49:24.885436 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.885439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.885627 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:49:24.886669 (+ 1042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.886670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.886685 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.886737 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.887376 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.887869 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.888884 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.889079 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.889162 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.891614 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.891615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.891616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.891622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.893211 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.893576 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.893577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.893577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.893629 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.893789 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.893790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.893790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.893893 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.893943 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.894133 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.894189 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:24.894507 6431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.885357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38977 (local address 127.0.98.3:58131)
0925 14:49:24.885499 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:24.885544 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:24.886554 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:24.886807 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:24.886811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:24.887073 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:24.887254 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:24.887258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.887973 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.887973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:24.888773 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:24.888775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.889171 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.889172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:24.889301 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.889863 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:24.889900 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:24.891508 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:24.893284 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:24.893291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:24.893293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:24.893521 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:24.893672 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:24.893673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:24.893673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:24.893755 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:24.893935 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:24.893937 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:24.894045 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:49:24.894122 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:49:24.894233 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260925 14:49:24.894980 6429 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38977
I20260925 14:49:24.895076 6429 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:24.895218 6429 heartbeater.cc:511] Master 127.0.98.62:38977 requested a full tablet report, sending...
I20260925 14:49:24.895522 5970 ts_manager.cc:194] Registered new tserver with Master: e4bf9862dc614fb2b73a7fae95ab2512 (127.0.98.3:39433)
I20260925 14:49:24.895895 5970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58131
I20260925 14:49:24.904278 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/test-admin.keytab.
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.919512 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:49:24.921221 6436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:24.920441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.0.98.1:43001)
0925 14:49:24.920618 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.920622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.920633 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:24.920659 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.920661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.920674 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.920742 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.921096 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35838: BlockingRecv error: recv got EOF from 127.0.0.1:35838 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
W20260925 14:49:24.921319 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.922282 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.923187 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:24.931032 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.923694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.0.98.1:43001)
0925 14:49:24.923742 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.923744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.923775 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:49:24.923861 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.923862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.923875 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.923972 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.924456 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.925174 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.925848 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.926162 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.926215 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.928380 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.928389 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.928389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.928395 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.929989 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.930301 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.930304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.930304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.930358 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.930578 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.930579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.930579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.930715 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.930817 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.930903 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.930962 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 14:49:24.932734 6436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:24.932268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.0.98.1:43001)
0925 14:49:24.932301 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.932302 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.932311 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:24.932374 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.932374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.932383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.932429 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.932664 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35858: BlockingRecv error: recv got EOF from 127.0.0.1:35858 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:49:24.932837 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.933744 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.934652 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:24.941594 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.935138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.0.98.1:43001)
0925 14:49:24.935188 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.935190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.935279 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:49:24.935350 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.935350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.935360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.935443 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.935791 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.936264 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.936953 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.937152 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.937237 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.939259 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.939261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.939261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.939267 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.940499 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.940845 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.940846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.940847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.940912 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.941109 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.941110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.941110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.941211 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.941306 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.941429 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.941503 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 14:49:24.943073 6436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:24.942510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.0.98.1:43001)
0925 14:49:24.942565 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.942567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.942633 (+ 66us) server_negotiation.cc:407] Connection header received
0925 14:49:24.942702 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.942703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.942712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.942765 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.943027 (+ 262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35872: BlockingRecv error: recv got EOF from 127.0.0.1:35872 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 14:49:24.943162 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790347739045519-392-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.944114 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:24.944953 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:49:24 dist-test-slave-z2s5 krb5kdc[5925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:49:24.951731 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:24.945442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.0.98.1:43001)
0925 14:49:24.945497 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:24.945499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:24.945552 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:49:24.945614 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:24.945614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:24.945624 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:24.945671 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:24.946020 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.946576 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.947266 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:24.947470 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:24.947547 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:24.949440 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:24.949442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:24.949442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:24.949448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:24.950644 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.950995 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.950997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.950997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.951052 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:24.951254 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:24.951255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:24.951255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:24.951354 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:24.951455 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:24.951560 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:49:24.951629 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:24.952394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6032
I20260925 14:49:24.957393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6166
I20260925 14:49:24.961853 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6300
I20260925 14:49:24.965998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 5938
2026-09-25T14:49:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1841 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:25 dist-test-slave-z2s5 krb5kdc[6449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:25 dist-test-slave-z2s5 krb5kdc[6449](info): set up 2 sockets
Sep 25 14:49:25 dist-test-slave-z2s5 krb5kdc[6449](info): commencing operation
krb5kdc: starting...
W20260925 14:49:26.030717 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 14:49:26.155000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46769
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:26.235647 6462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:26.235858 6462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:26.235881 6462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:26.237059 6462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:26.237106 6462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:26.237121 6462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:26.237134 6462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:26.237146 6462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:26.238680 6462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:26.238931 6462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:26.239179 6462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:26.244102 6467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:26.244210 6462 server_base.cc:1034] running on GCE node
W20260925 14:49:26.244107 6470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.244138 6468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:26.244637 6462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:26.244966 6462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:26.246161 6462 hybrid_clock.cc:648] HybridClock initialized: now 1790347766246138 us; error 38 us; skew 500 ppm
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:26.248981 6462 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 14:49:26.249578 6462 webserver.cc:460] Webserver started at http://127.0.98.62:37571/ using document root <none> and password file <none>
I20260925 14:49:26.249843 6462 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:26.249918 6462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:26.250078 6462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:26.251339 6462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "1be7adafed984ff798ec3ebe6345eb3e"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "04ab435e63ed8f75b35bd0d94969ccdf"
server_key_iv: "811b2b8b8366aa83c6a9ab87a17afd5f"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.251722 6462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "1be7adafed984ff798ec3ebe6345eb3e"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "04ab435e63ed8f75b35bd0d94969ccdf"
server_key_iv: "811b2b8b8366aa83c6a9ab87a17afd5f"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.253175 6462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:26.254069 6477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.254299 6462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:26.254433 6462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "1be7adafed984ff798ec3ebe6345eb3e"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "04ab435e63ed8f75b35bd0d94969ccdf"
server_key_iv: "811b2b8b8366aa83c6a9ab87a17afd5f"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.254528 6462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:26.265830 6462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:26.267961 6462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:26.268136 6462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:26.272590 6462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46769
I20260925 14:49:26.272639 6529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46769 every 8 connection(s)
I20260925 14:49:26.273044 6462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:26.274062 6530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:26.276337 6530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Bootstrap starting.
I20260925 14:49:26.276954 6530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:26.277271 6530 log.cc:824] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:26.278079 6530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: No bootstrap required, opened a new log
I20260925 14:49:26.280014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6462
I20260925 14:49:26.280117 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:26.279996 6530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260925 14:49:26.280241 6530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:26.280328 6530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1be7adafed984ff798ec3ebe6345eb3e, State: Initialized, Role: FOLLOWER
I20260925 14:49:26.280364 392 external_mini_cluster.cc:1428] Setting key 2e81697449c7a55f9971faf36343e6f5
I20260925 14:49:26.280503 6530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260925 14:49:26.280603 6530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:26.280645 6530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:26.280730 6530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:26.281494 6530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260925 14:49:26.281596 6530 leader_election.cc:302] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1be7adafed984ff798ec3ebe6345eb3e; no voters:
I20260925 14:49:26.281829 6530 leader_election.cc:288] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:26.281875 6535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:26.282021 6535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 LEADER]: Becoming Leader. State: Replica: 1be7adafed984ff798ec3ebe6345eb3e, State: Running, Role: LEADER
I20260925 14:49:26.282117 6535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } }
I20260925 14:49:26.282150 6530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:26.283553 6537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } }
I20260925 14:49:26.283701 6537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:26.284010 6549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:26.284010 6536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be7adafed984ff798ec3ebe6345eb3e. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } }
I20260925 14:49:26.284777 6536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:26.285025 6549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:26.286621 6549 catalog_manager.cc:1347] Generated new cluster ID: 85eb772cae084a0ba072ec26be539ce2
I20260925 14:49:26.286674 6549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:26.292224 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.281176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.0.98.62:46769)
0925 14:49:26.281776 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.281785 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.281839 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:49:26.281970 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.281979 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.282046 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.282987 (+ 941us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.283065 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.284708 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.285797 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.286376 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.286542 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.289012 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.289016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.289017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.289028 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.290797 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.291131 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.291133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.291134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.291234 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.291409 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.291410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.291410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.291685 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.291809 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.291901 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.292027 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":465,"threads_started":1}
I20260925 14:49:26.292966 6549 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:26.293459 6549 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:26.296725 6549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Generated new TSK 0
I20260925 14:49:26.296890 6549 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 14:49:26.322129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46769
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:26.409236 6556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:26.409435 6556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:26.409458 6556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:26.410653 6556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:26.410694 6556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:26.410723 6556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:26.412138 6556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:26.412384 6556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:26.412608 6556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:26.418551 6561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.418537 6562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.418560 6564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:26.418908 6556 server_base.cc:1034] running on GCE node
I20260925 14:49:26.419091 6556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:26.419348 6556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:26.420516 6556 hybrid_clock.cc:648] HybridClock initialized: now 1790347766420493 us; error 42 us; skew 500 ppm
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:26.423182 6556 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:49:26.423640 6556 webserver.cc:460] Webserver started at http://127.0.98.1:35871/ using document root <none> and password file <none>
I20260925 14:49:26.423869 6556 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:26.423913 6556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:26.424084 6556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:26.425151 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "fdfad53226de440e8e44f810d606b84d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "63ce56a772be2fd7268b0bacdcd2c560"
server_key_iv: "940403933a0e3b28025a19d820ed98d8"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.425526 6556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fdfad53226de440e8e44f810d606b84d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "63ce56a772be2fd7268b0bacdcd2c560"
server_key_iv: "940403933a0e3b28025a19d820ed98d8"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.426996 6556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:26.427837 6571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.428010 6556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:26.428120 6556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "fdfad53226de440e8e44f810d606b84d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "63ce56a772be2fd7268b0bacdcd2c560"
server_key_iv: "940403933a0e3b28025a19d820ed98d8"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.428202 6556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:26.445894 6556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:26.448292 6556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:26.448499 6556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:26.448793 6556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:26.449113 6556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:26.449169 6556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.449215 6556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:26.449237 6556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.455253 6556 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41367
I20260925 14:49:26.455303 6684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41367 every 8 connection(s)
I20260925 14:49:26.455626 6556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:26.457100 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6556
I20260925 14:49:26.457232 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:26.457489 392 external_mini_cluster.cc:1428] Setting key 49e47c8d589405fd0ca12186f6f8ef4a
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:49:26.466827 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.456399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46821 (local address 127.0.98.62:46769)
0925 14:49:26.456468 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.456470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.456679 (+ 209us) server_negotiation.cc:407] Connection header received
0925 14:49:26.457203 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.457204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.457220 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.457292 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.458629 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.459132 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.460728 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.460945 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.461034 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.463685 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.463687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.463688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.463697 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.465222 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.465732 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.465733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.465733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.465796 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.466062 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.466063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.466063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.466180 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.466260 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.466640 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.466733 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:49:26.467049 6687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.456376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:46821)
0925 14:49:26.456525 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.456568 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.457051 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.457384 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.457406 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.458172 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.458500 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.458506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.459239 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.459240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.460642 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.460645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.461045 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.461046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.461178 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.461827 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.461836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.463578 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.465376 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.465384 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.465386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.465637 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.465866 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.465867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.465868 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.465983 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.466249 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.466255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.466481 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.466617 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.466795 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260925 14:49:26.467626 6685 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46769
I20260925 14:49:26.467768 6685 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:26.467988 6685 heartbeater.cc:511] Master 127.0.98.62:46769 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260925 14:49:26.468627 6494 ts_manager.cc:194] Registered new tserver with Master: fdfad53226de440e8e44f810d606b84d (127.0.98.1:41367)
I20260925 14:49:26.469578 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 14:49:26.485996 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46769
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:26.565979 6690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:26.566242 6690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:26.566279 6690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:26.567492 6690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:26.567574 6690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:26.567619 6690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:26.569113 6690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:26.569407 6690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:26.569648 6690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:26.574831 6695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.574865 6696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.574885 6698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:26.575268 6690 server_base.cc:1034] running on GCE node
I20260925 14:49:26.575552 6690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:26.575778 6690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:26.576925 6690 hybrid_clock.cc:648] HybridClock initialized: now 1790347766576912 us; error 29 us; skew 500 ppm
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:26.579519 6690 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:49:26.580011 6690 webserver.cc:460] Webserver started at http://127.0.98.2:43971/ using document root <none> and password file <none>
I20260925 14:49:26.580245 6690 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:26.580314 6690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:26.580466 6690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:26.581511 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "43c36349d6824cedb50b403f3e96638f"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "f931949289c99d1aa5ebcccbea1c8ef0"
server_key_iv: "28b53e7044a543ce2267438c18cf4a30"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.581878 6690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43c36349d6824cedb50b403f3e96638f"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "f931949289c99d1aa5ebcccbea1c8ef0"
server_key_iv: "28b53e7044a543ce2267438c18cf4a30"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.583607 6690 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 14:49:26.584479 6705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.584750 6690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:26.584859 6690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "43c36349d6824cedb50b403f3e96638f"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "f931949289c99d1aa5ebcccbea1c8ef0"
server_key_iv: "28b53e7044a543ce2267438c18cf4a30"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.584950 6690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:26.594774 6690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:26.596812 6690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:26.596987 6690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:26.597218 6690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:26.597532 6690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:26.597591 6690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.597625 6690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:26.597652 6690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.603596 6690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35853
I20260925 14:49:26.603638 6818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35853 every 8 connection(s)
I20260925 14:49:26.603991 6690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:26.610991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6690
I20260925 14:49:26.611101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:26.611291 392 external_mini_cluster.cc:1428] Setting key d31bbeb8a3e3b7308fc1e6e1c036a4da
I20260925 14:49:26.613186 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.604762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34321 (local address 127.0.98.62:46769)
0925 14:49:26.604823 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.604825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.605046 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:49:26.605502 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.605503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.605512 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.605571 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.606537 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.607019 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.607991 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.608179 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.608234 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.610230 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.610232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.610232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.610238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.611837 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.612254 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.612255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.612256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.612315 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.612559 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.612560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.612560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.612673 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.612753 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.613025 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.613098 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:26.613386 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.604740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:34321)
0925 14:49:26.604909 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.604958 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.605422 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.605622 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.605625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.606203 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.606425 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.606430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.607106 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.607107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.607905 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.607907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.608252 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.608253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.608352 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.608794 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.608802 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.610113 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.611933 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.611940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.611942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.612159 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.612385 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.612385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.612386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.612492 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.612744 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.612750 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.612893 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.612994 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.613148 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260925 14:49:26.613839 6819 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46769
I20260925 14:49:26.613941 6819 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:26.614138 6819 heartbeater.cc:511] Master 127.0.98.62:46769 requested a full tablet report, sending...
I20260925 14:49:26.614537 6494 ts_manager.cc:194] Registered new tserver with Master: 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:26.614919 6494 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34321
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 14:49:26.637661 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46769
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:26.720525 6824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:26.720778 6824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:26.720819 6824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:26.721947 6824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:26.722018 6824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:26.722062 6824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:26.723495 6824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:26.723755 6824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:26.723999 6824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:26.729293 6832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.729259 6830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.729246 6829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:26.729400 6824 server_base.cc:1034] running on GCE node
I20260925 14:49:26.729665 6824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:26.729923 6824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:26.731083 6824 hybrid_clock.cc:648] HybridClock initialized: now 1790347766731079 us; error 38 us; skew 500 ppm
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:26.733652 6824 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:49:26.734074 6824 webserver.cc:460] Webserver started at http://127.0.98.3:39489/ using document root <none> and password file <none>
I20260925 14:49:26.734306 6824 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:26.734376 6824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:26.734541 6824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:26.735574 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "6b61378a78c244f6ad71c8477c0c2acb"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "9d3984c5f54d5e0d178497e08241b047"
server_key_iv: "2a0d29920500a8350283e3d613f04067"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.735960 6824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6b61378a78c244f6ad71c8477c0c2acb"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "9d3984c5f54d5e0d178497e08241b047"
server_key_iv: "2a0d29920500a8350283e3d613f04067"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.737331 6824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:26.738056 6839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.738258 6824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:26.738354 6824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "6b61378a78c244f6ad71c8477c0c2acb"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "9d3984c5f54d5e0d178497e08241b047"
server_key_iv: "2a0d29920500a8350283e3d613f04067"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.738457 6824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:26.753077 6824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:26.755365 6824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:26.755538 6824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:26.755801 6824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:26.756126 6824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:26.756183 6824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.756228 6824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:26.756318 6824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.762624 6824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41305
I20260925 14:49:26.762708 6952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41305 every 8 connection(s)
I20260925 14:49:26.763051 6824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:26.772192 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.763778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40871 (local address 127.0.98.62:46769)
0925 14:49:26.763840 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.763842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.764077 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:49:26.764540 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.764541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.764551 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.764614 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.765509 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.766042 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.767076 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.767275 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.767336 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.769403 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.769404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.769405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.769411 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.770853 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.771314 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.771315 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.771315 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.771380 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.771629 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.771630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.771630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.771749 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.771823 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.772059 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.772125 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:49:26.772338 6955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.763755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.3:40871)
0925 14:49:26.763947 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.763989 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.764457 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.764666 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.764670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.765232 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.765422 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.765427 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.766134 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.766135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.766999 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.767001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.767350 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.767350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.767450 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.767903 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.767911 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.769262 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.770925 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.770930 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.770932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.771228 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.771452 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.771453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.771453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.771549 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.771824 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.771828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.771948 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.772030 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.772144 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260925 14:49:26.772735 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6824
I20260925 14:49:26.772854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:26.772833 6953 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46769
I20260925 14:49:26.772930 6953 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:26.773082 392 external_mini_cluster.cc:1428] Setting key b713aeefdf6774273daebdcaa86b9a6d
I20260925 14:49:26.773104 6953 heartbeater.cc:511] Master 127.0.98.62:46769 requested a full tablet report, sending...
I20260925 14:49:26.773532 6493 ts_manager.cc:194] Registered new tserver with Master: 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305)
I20260925 14:49:26.774021 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40871
I20260925 14:49:26.774055 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260925 14:49:26.800901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45995
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46769,127.0.98.61:45995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:26.885900 6958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:26.886102 6958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:26.886125 6958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:26.887260 6958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:26.887300 6958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:26.887315 6958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:26.887326 6958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:26.887338 6958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:26.888717 6958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46769,127.0.98.61:45995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
W20260925 14:49:26.888815 6958 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:46769,127.0.98.61:45995'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:49:26.888967 6958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:26.889168 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:26.894016 6964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.894016 6963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:26.894018 6966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:26.894368 6958 server_base.cc:1034] running on GCE node
I20260925 14:49:26.894675 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:26.895030 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:26.896217 6958 hybrid_clock.cc:648] HybridClock initialized: now 1790347766896203 us; error 44 us; skew 500 ppm
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:26.899384 6958 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 14:49:26.899940 6958 webserver.cc:460] Webserver started at http://127.0.98.61:32777/ using document root <none> and password file <none>
I20260925 14:49:26.900168 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:26.900213 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:26.900367 6958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:26.901437 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/instance:
uuid: "242f729351b44efa9652bb5f9f867b0d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "dc5814633ed9579d2a40b76422acee65"
server_key_iv: "e78d355e696665a291b75785dd582b72"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.901803 6958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal/instance:
uuid: "242f729351b44efa9652bb5f9f867b0d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "dc5814633ed9579d2a40b76422acee65"
server_key_iv: "e78d355e696665a291b75785dd582b72"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.903221 6958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:26.903981 6973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.904165 6958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:26.904254 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
uuid: "242f729351b44efa9652bb5f9f867b0d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "dc5814633ed9579d2a40b76422acee65"
server_key_iv: "e78d355e696665a291b75785dd582b72"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.904366 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:26.914353 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:26.916657 6958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:26.916815 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:26.920732 6958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45995
I20260925 14:49:26.920761 7025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45995 every 8 connection(s)
I20260925 14:49:26.921128 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:49:26.922067 7026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:26.923647 7026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:49:26.925823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 6958
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:26.933876 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.924367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49004 (local address 127.0.98.62:46769)
0925 14:49:26.924450 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.924452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.924650 (+ 198us) server_negotiation.cc:407] Connection header received
0925 14:49:26.925156 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.925157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.925170 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.925224 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.925952 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.926774 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.928143 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.928435 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.928517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.930901 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.930903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.930903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.930912 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.932455 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.932852 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.932853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.932853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.932915 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.933155 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.933156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.933156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.933354 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.933431 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.933702 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.933781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:26.934087 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.924344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49004)
0925 14:49:26.924522 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.924557 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.925040 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.925292 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.925296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.925546 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.925816 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.925822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.926892 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.926893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.928049 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.928051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.928555 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.928556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.928689 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.929262 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.929272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.930814 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.932542 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.932547 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.932549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.932769 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.932979 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.932980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.932981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.933084 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.933434 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.933440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.933568 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.933672 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.933836 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260925 14:49:26.934773 7026 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:49:26.935678 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.926365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.0.98.61:45995)
0925 14:49:26.926598 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.926603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.926627 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:49:26.926648 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.926650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.926676 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.926775 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.927572 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.928476 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.929419 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.929705 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.929779 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.932451 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.932459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.932459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.932466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.934077 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.934673 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.934676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.934676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.934746 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.934965 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.934966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.934966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.935148 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.935290 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.935492 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.935571 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":149,"threads_started":1}
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:26.941633 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.935036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60600)
0925 14:49:26.935083 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.935094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.935132 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.935358 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.935358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.935490 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.935534 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.935535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.936301 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.936301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.936760 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.936762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.937087 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.937087 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.937136 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.937540 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.937552 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.938676 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.940638 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.940639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.940639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.940849 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.941052 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.941053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.941053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.941107 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.941333 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.941334 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.941371 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.941439 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.941530 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 14:49:26.941712 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.935071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.98.61:45995)
0925 14:49:26.935221 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.935222 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.935229 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:26.935278 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.935278 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.935285 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.935356 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.935689 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.936213 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.936829 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.937027 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.937079 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.938791 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.938792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.938792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.938799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.940534 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.940936 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.940937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.940938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.940992 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.941165 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.941166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.941166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.941272 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.941342 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.941467 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.941580 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260925 14:49:26.943022 7026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Bootstrap starting.
I20260925 14:49:26.943672 7026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:26.943956 7026 log.cc:824] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:26.944618 7026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: No bootstrap required, opened a new log
I20260925 14:49:26.945959 7026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } }
I20260925 14:49:26.946079 7026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:26.946125 7026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 242f729351b44efa9652bb5f9f867b0d, State: Initialized, Role: FOLLOWER
I20260925 14:49:26.946260 7026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } }
I20260925 14:49:26.946550 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } } }
I20260925 14:49:26.946609 7026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:26.946709 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:49:26.948166 7044 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:49:26.948220 7044 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:49:26.949677 6958 master_runner.cc:428] Detected that this master 242f729351b44efa9652bb5f9f867b0d is joining an existing cluster
I20260925 14:49:26.949724 6958 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:45995
I20260925 14:49:26.951506 7033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:26.951581 7033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } }
I20260925 14:49:26.951773 7033 leader_election.cc:288] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769)
I20260925 14:49:26.951977 6504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "242f729351b44efa9652bb5f9f867b0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1be7adafed984ff798ec3ebe6345eb3e" is_pre_election: true
I20260925 14:49:26.952314 6975 leader_election.cc:302] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 242f729351b44efa9652bb5f9f867b0d; no voters: 1be7adafed984ff798ec3ebe6345eb3e
I20260925 14:49:26.952445 7033 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 14:49:26.957798 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.950891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49012 (local address 127.0.98.62:46769)
0925 14:49:26.950947 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.950949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.951171 (+ 222us) server_negotiation.cc:407] Connection header received
0925 14:49:26.951246 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.951247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.951256 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.951309 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.951712 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.952506 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.953587 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.953896 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.953938 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.954994 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.954995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.954995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.955002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.956382 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.956902 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.956903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.956904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.956966 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.957218 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.957220 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.957220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.957331 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.957406 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.957592 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.957691 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:49:26.957799 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.950807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60604 (local address 127.0.98.61:45995)
0925 14:49:26.950853 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.950855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.951015 (+ 160us) server_negotiation.cc:407] Connection header received
0925 14:49:26.951088 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.951089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.951098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.951171 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.951977 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.952657 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.953341 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.953567 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.953637 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.954714 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.954715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.954715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.954723 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.956396 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.956884 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.956886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.956886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.956945 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.957196 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.957197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.957198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.957419 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.957492 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.957658 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.957716 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:26.957784 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.950744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49012)
0925 14:49:26.951122 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.951130 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.951192 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.951314 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.951314 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.951531 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.951599 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.951601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.952582 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.952583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.953477 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.953479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.954015 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.954016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.954080 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.954452 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.954457 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.954884 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.956504 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.956504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.956505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.956811 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.957061 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.957061 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.957061 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.957139 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.957397 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.957398 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.957456 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.957553 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.957658 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":75,"threads_started":1}
I20260925 14:49:26.958485 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.950739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60604)
0925 14:49:26.950956 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.950967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.951028 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.951173 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.951174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.951442 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.951879 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.951881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.952733 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.952733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.953257 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.953258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.953659 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.953660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.953715 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.954181 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.954186 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.954614 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.956487 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.956488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.956489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.956811 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.957053 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.957054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.957054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.957127 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.957486 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.957486 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.957555 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.957808 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.958356 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"spinlock_wait_cycles":9600,"thread_start_us":100,"threads_started":1}
Sep 25 14:49:26 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:26.967216 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.960077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60616 (local address 127.0.98.61:45995)
0925 14:49:26.960114 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.960116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.960263 (+ 147us) server_negotiation.cc:407] Connection header received
0925 14:49:26.960353 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.960354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.960364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.960419 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.960820 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.961259 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.962015 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.962213 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.962294 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.964245 (+ 1951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.964246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.964246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.964253 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.965588 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.966430 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.966431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.966432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.966491 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.966697 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.966698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.966698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.966802 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.966872 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.967053 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.967125 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:26.967249 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.960022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60616)
0925 14:49:26.960174 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.960204 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.960288 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.960435 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.960439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.960690 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.960733 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.960735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.961338 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.961339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.961920 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.961922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.962299 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.962300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.962441 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.962887 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.962891 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.964160 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.965692 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.965698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.965700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.966311 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.966559 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.966561 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.966561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.966620 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.966879 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.966886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.966934 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.967028 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.967139 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":68,"threads_started":1}
I20260925 14:49:26.967741 6493 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:45995: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "242f729351b44efa9652bb5f9f867b0d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 45995
}
attrs {
promote: true
}
}
dest_uuid: "1be7adafed984ff798ec3ebe6345eb3e"
cas_config_opid_index: -1
I20260925 14:49:26.968052 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } }
I20260925 14:49:26.968514 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 242f729351b44efa9652bb5f9f867b0d (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:26.968816 7000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:26.969558 7054 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:45995
I20260925 14:49:26.969700 7000 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Refusing update from remote peer 1be7adafed984ff798ec3ebe6345eb3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:49:26.969785 6958 master.cc:537] Master@127.0.98.61:45995 shutting down...
I20260925 14:49:26.969794 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:26.969981 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:26.970021 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:49:26.970773 6480 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 14:49:26.971168 6480 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e -> Peer 242f729351b44efa9652bb5f9f867b0d (127.0.98.61:45995): Couldn't send request to peer 242f729351b44efa9652bb5f9f867b0d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:49:26.972451 6958 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:26.972528 6958 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:26.972563 6958 tablet_replica.cc:330] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: stopping tablet replica
I20260925 14:49:26.985219 6958 master.cc:552] Master@127.0.98.61:45995 shutdown complete.
I20260925 14:49:26.985266 6958 master_runner.cc:305] Clearing existing system tablet
I20260925 14:49:26.985407 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:26.985455 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:26.986083 6958 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:26.986889 7058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:26.987035 6958 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:26.987093 6958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:49:26.987118 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
uuid: "242f729351b44efa9652bb5f9f867b0d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "dc5814633ed9579d2a40b76422acee65"
server_key_iv: "e78d355e696665a291b75785dd582b72"
server_key_version: "encryptionkey@0"
I20260925 14:49:26.987735 6958 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:26.989398 6958 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:49:26.989490 6958 log.cc:1196] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:49:26.989684 6958 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Deleting consensus metadata
I20260925 14:49:26.989815 6958 master_runner.cc:315] Copying system tablet from 127.0.98.62:46769
I20260925 14:49:26.990435 6958 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46769
I20260925 14:49:26.996670 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.990827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49024 (local address 127.0.98.62:46769)
0925 14:49:26.990874 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:26.990876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:26.990942 (+ 66us) server_negotiation.cc:407] Connection header received
0925 14:49:26.991017 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:26.991017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:26.991026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:26.991076 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:26.991436 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.991902 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.992935 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.993107 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.993157 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.994186 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:26.994187 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:26.994187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:26.994194 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:26.995625 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.996011 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.996012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.996012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.996061 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:26.996252 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:26.996252 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:26.996253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:26.996347 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:26.996395 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:26.996548 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:49:26.996606 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:49:26.996748 7064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:26.990734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49024)
0925 14:49:26.990866 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:26.990878 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:26.990953 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:26.991083 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:26.991084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:26.991284 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:26.991355 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.991357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.991978 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.991979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.992853 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:26.992856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:26.993175 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:26.993176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:26.993243 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:26.993687 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:26.993692 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:26.994097 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:26.995706 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.995707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.995708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.995935 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.996127 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:26.996128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:26.996128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:26.996191 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:26.996406 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:26.996407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:26.996453 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:49:26.996525 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:49:26.996623 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260925 14:49:26.997040 6514 tablet_copy_service.cc:140] P 1be7adafed984ff798ec3ebe6345eb3e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 242f729351b44efa9652bb5f9f867b0d ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49024)
I20260925 14:49:26.997117 6514 tablet_copy_service.cc:161] P 1be7adafed984ff798ec3ebe6345eb3e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 242f729351b44efa9652bb5f9f867b0d at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49024: session id = 242f729351b44efa9652bb5f9f867b0d-00000000000000000000000000000000
I20260925 14:49:26.997956 6514 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:49:26.998642 6958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:49:26.999643 6958 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: tablet copy: Starting download of 0 data blocks...
I20260925 14:49:26.999828 6958 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: tablet copy: Starting download of 1 WAL segments...
I20260925 14:49:27.000744 6958 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:49:27.001565 6514 tablet_copy_service.cc:342] P 1be7adafed984ff798ec3ebe6345eb3e: Request end of tablet copy session 242f729351b44efa9652bb5f9f867b0d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:49024
I20260925 14:49:27.001621 6514 tablet_copy_service.cc:433] P 1be7adafed984ff798ec3ebe6345eb3e: ending tablet copy session 242f729351b44efa9652bb5f9f867b0d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 242f729351b44efa9652bb5f9f867b0d
I20260925 14:49:27.004176 6958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:27.005726 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.005869 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.006031 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.006043 6958 server_base.cc:1034] running on GCE node
I20260925 14:49:27.006230 6958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:27.006438 6958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:27.007575 6958 hybrid_clock.cc:648] HybridClock initialized: now 1790347767007564 us; error 32 us; skew 500 ppm
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:27.008983 6958 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260925 14:49:27.009275 6958 webserver.cc:460] Webserver started at http://127.0.98.61:41249/ using document root <none> and password file <none>
I20260925 14:49:27.009394 6958 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.009447 6958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.010052 6958 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:27.010583 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.010727 6958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:27.010792 6958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
uuid: "242f729351b44efa9652bb5f9f867b0d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "dc5814633ed9579d2a40b76422acee65"
server_key_iv: "e78d355e696665a291b75785dd582b72"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.010849 6958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.041657 6958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:27.043483 6958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:27.046572 6958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45995
I20260925 14:49:27.046605 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45995 every 8 connection(s)
I20260925 14:49:27.046936 6958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:49:27.047653 7128 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:49:27.048048 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Bootstrap starting.
I20260925 14:49:27.050575 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:49:27.050832 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Bootstrap complete.
I20260925 14:49:27.051040 7128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } }
I20260925 14:49:27.051126 7128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 242f729351b44efa9652bb5f9f867b0d, State: Initialized, Role: LEARNER
I20260925 14:49:27.051177 7128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } }
I20260925 14:49:27.051364 7128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:27.051330 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:27.052724 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: LEARNER
I20260925 14:49:27.053603 7142 catalog_manager.cc:1259] Loaded cluster ID: 85eb772cae084a0ba072ec26be539ce2
I20260925 14:49:27.053640 7142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: loading cluster ID for follower catalog manager: success
I20260925 14:49:27.054462 7142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: acquiring CA information for follower catalog manager: success
I20260925 14:49:27.054741 7142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:49:27.094332 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.087775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.0.98.61:45995)
0925 14:49:27.087925 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.087927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.087934 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:27.087956 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.087957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.087965 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.088041 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.088675 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.089395 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.090124 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.090342 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.090445 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.091691 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.091692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.091692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.091699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.093157 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.093520 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.093521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.093522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.093581 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.093790 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.093790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.093791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.093905 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.094009 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.094136 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.094220 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260925 14:49:27.121189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35907
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:27.219105 7146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:27.219333 7146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:27.219379 7146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:27.220535 7146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:27.220602 7146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:27.220629 7146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:27.220660 7146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:27.220711 7146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:27.222271 7146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:27.222569 7146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:27.222790 7146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:27.229332 7151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.229334 7154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.229332 7152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.229738 7146 server_base.cc:1034] running on GCE node
I20260925 14:49:27.229984 7146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:27.230289 7146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:27.231452 7146 hybrid_clock.cc:648] HybridClock initialized: now 1790347767231462 us; error 72 us; skew 500 ppm
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:27.234061 7146 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 14:49:27.234654 7146 webserver.cc:460] Webserver started at http://127.0.98.60:41337/ using document root <none> and password file <none>
I20260925 14:49:27.234918 7146 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.234964 7146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.235111 7146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:27.236258 7146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/instance:
uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
format_stamp: "Formatted at 2026-09-25 14:49:27 on dist-test-slave-z2s5"
server_key: "56b856f30d775c7ddd467580bcc0ab65"
server_key_iv: "690c637ce6615ae9be075a5a2d1ddfd2"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.236649 7146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal/instance:
uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
format_stamp: "Formatted at 2026-09-25 14:49:27 on dist-test-slave-z2s5"
server_key: "56b856f30d775c7ddd467580bcc0ab65"
server_key_iv: "690c637ce6615ae9be075a5a2d1ddfd2"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.238030 7146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:27.238945 7161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.239097 7146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.239178 7146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
format_stamp: "Formatted at 2026-09-25 14:49:27 on dist-test-slave-z2s5"
server_key: "56b856f30d775c7ddd467580bcc0ab65"
server_key_iv: "690c637ce6615ae9be075a5a2d1ddfd2"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.239252 7146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.259790 7146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:27.262682 7146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:27.262835 7146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:27.266806 7146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35907
I20260925 14:49:27.266875 7213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35907 every 8 connection(s)
I20260925 14:49:27.267230 7146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:49:27.268167 7214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:27.269766 7214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:27.276438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 7146
I20260925 14:49:27.279544 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.270665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49040 (local address 127.0.98.62:46769)
0925 14:49:27.270729 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.270732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.270841 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:49:27.271289 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.271290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.271299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.271357 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.271933 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.272437 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.273501 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.273692 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.273778 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.275913 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.275915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.275915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.275922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.277886 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.278358 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.278359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.278359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.278462 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.278736 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.278737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.278738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.278869 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.278922 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.279340 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.279455 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:49:27.279685 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.270558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49040)
0925 14:49:27.270712 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.270753 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.271207 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.271427 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.271430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.271662 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.271845 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.271849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.272548 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.272549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.273354 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.273356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.273785 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.273786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.273905 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.274484 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.274492 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.275818 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.277965 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.277969 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.277970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.278282 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.278524 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.278524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.278525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.278657 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.279000 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.279005 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.279186 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.279322 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.279485 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260925 14:49:27.280663 7214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:27.286974 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.277059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.0.98.60:35907)
0925 14:49:27.277227 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.277231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.277249 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:49:27.277266 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.277267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.277278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.277343 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.278057 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.279297 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.280255 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.280736 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.280811 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.283416 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.283424 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.283424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.283431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.285294 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.285687 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.285689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.285690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.285762 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.285997 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.285998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.285998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.286138 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.286748 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.286818 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.286904 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260925 14:49:27.287744 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.280908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60656)
0925 14:49:27.280958 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.280968 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.281002 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.281152 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.281153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.281285 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.281323 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.281324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.282005 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.282006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.282800 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.282803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.283193 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.283194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.283270 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.283657 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.283661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.284845 (+ 1184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.286716 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.286718 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.286718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.286911 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.287118 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.287118 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.287119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.287218 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.287485 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.287486 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.287518 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.287598 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.287688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:49:27.287847 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.280986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60656 (local address 127.0.98.61:45995)
0925 14:49:27.281033 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.281035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.281045 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:27.281068 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.281069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.281088 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.281150 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.281408 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.281883 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.282880 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.283121 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.283176 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.284926 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.284928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.284928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.284935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.286636 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.286999 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.287000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.287001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.287057 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.287275 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.287276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.287276 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.287409 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.287488 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.287642 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.287729 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:49:27.288187 7214 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:49:27.293958 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.288487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44946 (local address 127.0.98.60:35907)
0925 14:49:27.288518 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.288519 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.288528 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:27.288584 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.288585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.288593 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.288642 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.288829 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.289260 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.289849 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.290034 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.290088 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.291730 (+ 1642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.291731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.291731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.291737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.293105 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.293381 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.293381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.293382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.293427 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.293567 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.293567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.293567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.293647 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.293690 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.293790 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.293879 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":14}
I20260925 14:49:27.293969 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.288439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.0.1:44946)
0925 14:49:27.288489 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.288500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.288534 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.288647 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.288647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.288749 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.288775 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.288776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.289320 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.289321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.289774 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.289775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.290100 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.290101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.290135 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.290606 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.290609 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.291649 (+ 1040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.293175 (+ 1526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.293176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.293176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.293316 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.293484 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.293484 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.293484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.293530 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.293695 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.293696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.293725 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.293773 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.293838 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:49:27.295361 7214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Bootstrap starting.
I20260925 14:49:27.296178 7214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:27.296468 7214 log.cc:824] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:27.297183 7214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: No bootstrap required, opened a new log
I20260925 14:49:27.298655 7214 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } }
I20260925 14:49:27.298781 7214 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:27.298804 7214 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4b1e17e0c4b417bb5730eb1641238aa, State: Initialized, Role: FOLLOWER
I20260925 14:49:27.298945 7214 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } }
I20260925 14:49:27.299249 7214 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:27.299214 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } } }
I20260925 14:49:27.299386 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:49:27.301034 7230 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:49:27.301095 7230 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:49:27.303253 7146 master_runner.cc:428] Detected that this master e4b1e17e0c4b417bb5730eb1641238aa is joining an existing cluster
I20260925 14:49:27.303290 7146 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:35907
I20260925 14:49:27.330703 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.304372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.0.98.62:46769)
0925 14:49:27.304419 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.304421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.304533 (+ 112us) server_negotiation.cc:407] Connection header received
0925 14:49:27.304563 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.304564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.304575 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.304633 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.305249 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.306068 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307194 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.307492 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307532 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.327400 (+ 19868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.327402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.327402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.327411 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.329270 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.329757 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.329758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.329759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.329818 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330026 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.330027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.330027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.330182 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.330224 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.330485 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.330576 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:49:27.330796 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.304685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.0.98.61:45995)
0925 14:49:27.304728 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.304742 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.304749 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:27.304770 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.304771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.304782 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.304834 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.305564 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.306277 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307235 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.307532 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307571 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.327956 (+ 20385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.327958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.327958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.327967 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.329526 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.329944 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.329945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.329946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.329998 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330209 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.330210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.330210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.330347 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.330427 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.330617 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.330694 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:27.331106 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.304515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60658)
0925 14:49:27.304626 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.304635 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.304668 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.305246 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.305247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.305423 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.305473 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.305475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.306361 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.306361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.307153 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.307155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.307638 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.307690 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.327088 (+ 19398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.327096 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.327846 (+ 750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.329620 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.329620 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.329621 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.329862 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.330069 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.330070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.330116 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.330473 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.330474 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.330509 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.330896 (+ 387us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.330999 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260925 14:49:27.331420 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.304282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49056)
0925 14:49:27.304434 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.304446 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.304510 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.304880 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.304881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.305102 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.305156 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.305158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.306154 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.306155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.307097 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.307099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.307608 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.307671 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.308357 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.308363 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.327286 (+ 18923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.329361 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.329363 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.329363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.329666 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.329895 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.329896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.329896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.329952 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.330284 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.330285 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.330339 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.331115 (+ 776us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.331333 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260925 14:49:27.331847 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.304860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.98.60:35907)
0925 14:49:27.304891 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.304893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.304903 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:27.304920 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.304920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.304929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.304996 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.305363 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.306077 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307129 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.307406 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307462 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.327311 (+ 19849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.327312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.327313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.327319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.330129 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330505 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.330506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.330507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.330551 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330708 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.330709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.330709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.331024 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.331082 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.331694 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.331769 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:49:27.332005 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.304710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.0.1:44948)
0925 14:49:27.304833 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.304842 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.304884 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.304997 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.304998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.305215 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.305282 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.305283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.306161 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.306162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.307043 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.307046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.307515 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.307516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.307569 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.307991 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.307996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.327200 (+ 19204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.330216 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330216 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.330217 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.330437 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.330608 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.330608 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.330609 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.330652 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.331102 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.331103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.331136 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.331299 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.331390 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347766, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:49:27.338231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6556
I20260925 14:49:27.341436 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.333525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.0.1:44960)
0925 14:49:27.333573 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.333583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.333618 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.333740 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.333740 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.333900 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.333957 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.333959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.334650 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.334651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.335478 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.335481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.335836 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.335837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.335906 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.336331 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.336336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.337672 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.340215 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.340216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.340216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.340458 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.340669 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.340670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.340671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.340731 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.341065 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.341066 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.341114 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.341161 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.341281 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:49:27.341802 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.333578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.0.98.60:35907)
0925 14:49:27.333610 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.333612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.333620 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:27.333678 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.333679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.333686 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.333727 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.334020 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.334569 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.335574 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.335760 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.335827 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.337774 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.337775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.337776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.337783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.340118 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.340544 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.340544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.340545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.340601 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.340810 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.340811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.340811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.340993 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.341548 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.341596 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.341670 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:49:27.342301 6493 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:35907: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 35907
}
attrs {
promote: true
}
}
dest_uuid: "1be7adafed984ff798ec3ebe6345eb3e"
cas_config_opid_index: 5
I20260925 14:49:27.342587 6493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } }
I20260925 14:49:27.343271 7054 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.344197 7054 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:35907
I20260925 14:49:27.344269 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.344302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41367
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46769
--webserver_port=35871
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:49:27.344334 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.345191 7188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:27.346048 7188 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 FOLLOWER]: Refusing update from remote peer 1be7adafed984ff798ec3ebe6345eb3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:49:27.346310 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:27.347788 7188 raft_consensus.cc:858] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } }
W20260925 14:49:27.347858 7188 raft_consensus.cc:1565] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } }
I20260925 14:49:27.348039 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.344225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60664)
0925 14:49:27.344308 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.344318 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.344359 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.344693 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.344694 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.344697 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.344735 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.344737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.345652 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.345653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.346887 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.346890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.347779 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.347780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.347846 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.347868 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.347919 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.347970 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 14:49:27.348219 7146 master.cc:537] Master@127.0.98.60:35907 shutting down...
I20260925 14:49:27.348434 7188 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 242f729351b44efa9652bb5f9f867b0d (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:27.348573 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.344445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.0.98.61:45995)
0925 14:49:27.344497 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.344499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.344509 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:27.344534 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.344535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.344558 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.344630 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.344804 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.345560 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.347000 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.347594 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.347663 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.347745 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.348439 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.348489 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:49:27.348771 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 LEARNER]: Refusing update from remote peer 1be7adafed984ff798ec3ebe6345eb3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:49:27.348979 7053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:49:27.349529 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.349586 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be7adafed984ff798ec3ebe6345eb3e. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:27.349665 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 14:49:27.349704 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:27.349741 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 14:49:27.349777 7219 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } }
I20260925 14:49:27.349810 7219 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260925 14:49:27.350872 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be7adafed984ff798ec3ebe6345eb3e. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.351002 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: LEARNER
I20260925 14:49:27.351038 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.351084 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: LEARNER
I20260925 14:49:27.351722 7239 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: attempting to promote NON_VOTER 242f729351b44efa9652bb5f9f867b0d to VOTER
I20260925 14:49:27.351948 7239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } }
I20260925 14:49:27.351737 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: true } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.352099 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
W20260925 14:49:27.352645 6478 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e -> Peer e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60:35907): Couldn't send request to peer e4b1e17e0c4b417bb5730eb1641238aa. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:49:27.353065 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 LEARNER]: Refusing update from remote peer 1be7adafed984ff798ec3ebe6345eb3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:49:27.353263 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:49:27.353582 7146 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260925 14:49:27.353660 7146 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260925 14:49:27.353700 7146 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: stopping tablet replica
I20260925 14:49:27.354035 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 242f729351b44efa9652bb5f9f867b0d (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.354230 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 242f729351b44efa9652bb5f9f867b0d (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.355147 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.355247 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.355284 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.355357 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.355437 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.355516 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:49:27.370059 7146 master.cc:552] Master@127.0.98.60:35907 shutdown complete.
I20260925 14:49:27.370121 7146 master_runner.cc:305] Clearing existing system tablet
I20260925 14:49:27.370255 7146 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.370301 7146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.371088 7146 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:49:27.371645 7252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.371809 7146 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.371866 7146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:27.371889 7146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
format_stamp: "Formatted at 2026-09-25 14:49:27 on dist-test-slave-z2s5"
server_key: "56b856f30d775c7ddd467580bcc0ab65"
server_key_iv: "690c637ce6615ae9be075a5a2d1ddfd2"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.372553 7146 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:27.374068 7146 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:49:27.374135 7146 log.cc:1196] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:49:27.374361 7146 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Deleting consensus metadata
I20260925 14:49:27.374555 7146 master_runner.cc:315] Copying system tablet from 127.0.98.62:46769
I20260925 14:49:27.375136 7146 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:46769
I20260925 14:49:27.382458 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.375457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49064)
0925 14:49:27.375583 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.375594 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.375651 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.375810 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.375811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.375994 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.376048 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.376050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.377007 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.377008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.377808 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.377810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.378301 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.378301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.378365 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.378993 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.378999 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.379383 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.381266 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.381267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.381268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.381478 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.381684 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.381685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.381685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.381749 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.382088 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.382089 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.382138 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.382214 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.382328 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260925 14:49:27.382457 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.375534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.0.98.62:46769)
0925 14:49:27.375589 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.375592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.375638 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:49:27.375706 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.375708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.375718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.375798 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.376141 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.376900 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.377905 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.378215 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.378296 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.379481 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.379483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.379483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.379491 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.381182 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.381556 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.381557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.381557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.381616 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.381815 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.381816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.381816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.381955 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.382024 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.382260 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.382341 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:49:27.383045 6514 tablet_copy_service.cc:140] P 1be7adafed984ff798ec3ebe6345eb3e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4b1e17e0c4b417bb5730eb1641238aa ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49064)
I20260925 14:49:27.383111 6514 tablet_copy_service.cc:161] P 1be7adafed984ff798ec3ebe6345eb3e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4b1e17e0c4b417bb5730eb1641238aa at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49064: session id = e4b1e17e0c4b417bb5730eb1641238aa-00000000000000000000000000000000
I20260925 14:49:27.383687 6514 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:49:27.384150 7146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:49:27.385372 7146 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: tablet copy: Starting download of 0 data blocks...
I20260925 14:49:27.385528 7146 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: tablet copy: Starting download of 1 WAL segments...
I20260925 14:49:27.386395 7146 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:49:27.387346 6514 tablet_copy_service.cc:342] P 1be7adafed984ff798ec3ebe6345eb3e: Request end of tablet copy session e4b1e17e0c4b417bb5730eb1641238aa-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:49064
I20260925 14:49:27.387399 6514 tablet_copy_service.cc:433] P 1be7adafed984ff798ec3ebe6345eb3e: ending tablet copy session e4b1e17e0c4b417bb5730eb1641238aa-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4b1e17e0c4b417bb5730eb1641238aa
I20260925 14:49:27.392164 7146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:27.393378 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.393595 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.393654 7146 server_base.cc:1034] running on GCE node
W20260925 14:49:27.394888 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.395078 7146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:27.395211 7146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:27.396342 7146 hybrid_clock.cc:648] HybridClock initialized: now 1790347767396336 us; error 23 us; skew 500 ppm
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:27.397852 7146 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260925 14:49:27.398105 7146 webserver.cc:460] Webserver started at http://127.0.98.60:42413/ using document root <none> and password file <none>
I20260925 14:49:27.398216 7146 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.398252 7146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.398998 7146 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.399571 7269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.399686 7146 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.399727 7146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
format_stamp: "Formatted at 2026-09-25 14:49:27 on dist-test-slave-z2s5"
server_key: "56b856f30d775c7ddd467580bcc0ab65"
server_key_iv: "690c637ce6615ae9be075a5a2d1ddfd2"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.399788 7146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.429431 7146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:27.431408 7146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:27.434563 7146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35907
I20260925 14:49:27.434660 7321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35907 every 8 connection(s)
I20260925 14:49:27.434911 7146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:49:27.439163 7322 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:49:27.439656 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Bootstrap starting.
I20260925 14:49:27.442065 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:49:27.442283 7322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: Bootstrap complete.
I20260925 14:49:27.442519 7322 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } }
I20260925 14:49:27.442613 7322 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4b1e17e0c4b417bb5730eb1641238aa, State: Initialized, Role: LEARNER
I20260925 14:49:27.442656 7322 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } }
I20260925 14:49:27.442823 7322 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:49:27.444000 7240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:27.444263 7240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260925 14:49:27.444402 7339 catalog_manager.cc:1259] Loaded cluster ID: 85eb772cae084a0ba072ec26be539ce2
I20260925 14:49:27.444437 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.444535 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: LEARNER
W20260925 14:49:27.444738 7240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 14:49:27.444495 7339 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: loading cluster ID for follower catalog manager: success
I20260925 14:49:27.446233 7339 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: acquiring CA information for follower catalog manager: success
W20260925 14:49:27.446535 7240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260925 14:49:27.446566 7339 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:49:27.446588 7240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:27.446633 7240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:27.448243 7240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35871
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:27.448544 7240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:27.448792 7240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:27.455300 7342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.455432 7341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.455328 7344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.455726 7240 server_base.cc:1034] running on GCE node
I20260925 14:49:27.455947 7240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:27.456226 7240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:27.457418 7240 hybrid_clock.cc:648] HybridClock initialized: now 1790347767457400 us; error 31 us; skew 500 ppm
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:27.460163 7240 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:49:27.460664 7240 webserver.cc:460] Webserver started at http://127.0.98.1:35871/ using document root <none> and password file <none>
I20260925 14:49:27.460918 7240 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.461007 7240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.462998 7240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:27.463650 7351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.463822 7240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:27.463932 7240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "fdfad53226de440e8e44f810d606b84d"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "63ce56a772be2fd7268b0bacdcd2c560"
server_key_iv: "940403933a0e3b28025a19d820ed98d8"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.464303 7240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.473366 7240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:27.475284 7240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:27.475452 7240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:27.475677 7240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:27.475993 7240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:27.476051 7240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.476095 7240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:27.476140 7240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.482913 7240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41367
I20260925 14:49:27.482980 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41367 every 8 connection(s)
I20260925 14:49:27.483439 7240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:49:27.491963 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 7240
I20260925 14:49:27.492125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6690
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:27.496080 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.484445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35607 (local address 127.0.98.62:46769)
0925 14:49:27.484506 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.484509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.484718 (+ 209us) server_negotiation.cc:407] Connection header received
0925 14:49:27.485710 (+ 992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.485711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.485724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.485783 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.486841 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.487330 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.488499 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.488829 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.488888 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.491626 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.491627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.491627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.491634 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.494119 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.494705 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.494706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.494707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.494783 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.495305 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.495306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.495307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.495460 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.495526 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.495907 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.495990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:49:27.496316 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.484211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:35607)
0925 14:49:27.484580 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.484614 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.485598 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.485854 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.485859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.486374 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.486606 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.486612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.487443 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.487444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.488400 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.488402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.488977 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.488978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.489116 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.489782 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.489793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.491515 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.494209 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.494215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.494218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.494603 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.494860 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.494860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.494861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.495217 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.495554 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.495561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.495739 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.495871 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.496030 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":2,"thread_start_us":91,"threads_started":1}
I20260925 14:49:27.496742 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46769
I20260925 14:49:27.496863 7466 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.497051 7466 heartbeater.cc:511] Master 127.0.98.62:46769 requested a full tablet report, sending...
I20260925 14:49:27.497489 6493 ts_manager.cc:194] Re-registered known tserver with Master: fdfad53226de440e8e44f810d606b84d (127.0.98.1:41367)
I20260925 14:49:27.497623 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.484299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35735 (local address 127.0.98.60:35907)
0925 14:49:27.484894 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.484896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.484903 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:27.485783 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.485784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.485799 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.485870 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.487477 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.488490 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.489464 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.489674 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.489758 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.492961 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.492963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.492963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.492970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.494671 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.495053 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.495055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.495055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.495115 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.496980 (+ 1865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.496981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.496982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.497134 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.497193 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.497442 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.497512 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":57,"threads_started":1}
I20260925 14:49:27.498041 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35607
I20260925 14:49:27.498366 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.484206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.98.1:35735)
0925 14:49:27.484804 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.484812 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.485630 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.485894 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.485894 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.486380 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.486606 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.486612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.488598 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.488599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.489364 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.489366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.489755 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.489756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.489819 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.490224 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.490229 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.492862 (+ 2633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.494745 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.494746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.494747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.494964 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.495531 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.495532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.495533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.495624 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.497219 (+ 1595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.497219 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.497263 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.497333 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.498282 (+ 949us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":12,"spinlock_wait_cycles":7040,"thread_start_us":109,"threads_started":1}
I20260925 14:49:27.498508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35853
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46769
--webserver_port=43971
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:49:27.498787 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35907
I20260925 14:49:27.498827 7465 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.498872 7465 heartbeater.cc:511] Master 127.0.98.60:35907 requested a full tablet report, sending...
I20260925 14:49:27.499238 7286 ts_manager.cc:194] Registered new tserver with Master: fdfad53226de440e8e44f810d606b84d (127.0.98.1:41367)
I20260925 14:49:27.499563 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.484410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39305 (local address 127.0.98.61:45995)
0925 14:49:27.484508 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.484510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.484755 (+ 245us) server_negotiation.cc:407] Connection header received
0925 14:49:27.485757 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.485758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.485772 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.485810 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.486755 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.487513 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.488659 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.488942 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.489004 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.494662 (+ 5658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.494663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.494663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.494671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.496381 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.498532 (+ 2151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.498533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.498533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.498598 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.499006 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.499007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.499007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.499147 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.499200 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.499393 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.499479 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 14:49:27.499939 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.484208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.98.1:39305)
0925 14:49:27.484637 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.484646 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.485630 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.485897 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.485897 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.486340 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.486606 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.486611 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.487609 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.487610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.488576 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.488578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.489017 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.489018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.489117 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.490391 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.490396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.494554 (+ 4158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.496668 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.496669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.496669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.496894 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.498677 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.498678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.498679 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.498744 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.499233 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.499234 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.499275 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.499680 (+ 405us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.499800 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":6,"thread_start_us":105,"threads_started":1}
I20260925 14:49:27.500367 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45995
I20260925 14:49:27.500419 7467 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.500483 7467 heartbeater.cc:511] Master 127.0.98.61:45995 requested a full tablet report, sending...
I20260925 14:49:27.500854 7092 ts_manager.cc:194] Registered new tserver with Master: fdfad53226de440e8e44f810d606b84d (127.0.98.1:41367)
W20260925 14:49:27.589089 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:27.589318 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:27.589352 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:27.590596 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:27.590648 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:27.590701 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:27.592190 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43971
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:27.592453 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:27.592716 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:27.597839 7479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.597854 7478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.597854 7481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.598069 7473 server_base.cc:1034] running on GCE node
I20260925 14:49:27.598331 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:27.598593 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:27.599763 7473 hybrid_clock.cc:648] HybridClock initialized: now 1790347767599745 us; error 29 us; skew 500 ppm
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:27.602269 7473 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:49:27.602725 7473 webserver.cc:460] Webserver started at http://127.0.98.2:43971/ using document root <none> and password file <none>
I20260925 14:49:27.602994 7473 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.603068 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.604573 7473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:27.605103 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.605260 7473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:27.605398 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "43c36349d6824cedb50b403f3e96638f"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "f931949289c99d1aa5ebcccbea1c8ef0"
server_key_iv: "28b53e7044a543ce2267438c18cf4a30"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.605738 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.629235 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:27.631295 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:27.631456 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:27.631723 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:27.632040 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:27.632092 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.632136 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:27.632162 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.638293 7473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35853
I20260925 14:49:27.638337 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35853 every 8 connection(s)
I20260925 14:49:27.638743 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:27.645121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 7473
I20260925 14:49:27.645287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6824
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260925 14:49:27.651398 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.639517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37941 (local address 127.0.98.62:46769)
0925 14:49:27.639640 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.639642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.639871 (+ 229us) server_negotiation.cc:407] Connection header received
0925 14:49:27.640998 (+ 1127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.641000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.641011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.641044 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.641761 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.642554 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.643798 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.644092 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.644156 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.648113 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.648115 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.648115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.648122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.649786 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.650324 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.650325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.650325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.650383 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.650670 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.650672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.650672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.650797 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.650849 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.651198 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.651301 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 14:49:27.651430 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.639579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50463 (local address 127.0.98.60:35907)
0925 14:49:27.639714 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.639716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.639982 (+ 266us) server_negotiation.cc:407] Connection header received
0925 14:49:27.640986 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.640987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.641002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.641061 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.641753 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.642280 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.643450 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.643747 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.643812 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.648420 (+ 4608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.648422 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.648422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.648430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.649818 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.650323 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.650324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.650325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.650383 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.650873 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.650874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.650874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.651029 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.651100 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.651253 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.651337 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260925 14:49:27.651697 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.639521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:37941)
0925 14:49:27.639683 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.639724 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.640869 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.641137 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.641139 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.641479 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.641667 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.641668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.642747 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.642748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.643647 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.643648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.644206 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.644207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.644266 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.645801 (+ 1535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.645806 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.648017 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.649876 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.649882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.649885 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.650214 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.650463 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.650464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.650464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.650594 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.650862 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.650867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.651027 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.651165 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.651349 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260925 14:49:27.651808 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.639765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.98.2:50463)
0925 14:49:27.639905 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.639914 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.640881 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.641136 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.641139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.641429 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.641643 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.641647 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.642437 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.642437 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.643343 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.643345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.644072 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.644073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.644177 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.644953 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.644958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.648340 (+ 3382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.649938 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.649939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.649939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.650229 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.650488 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.650489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.650489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.650601 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.651093 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.651094 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.651136 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.651509 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.651630 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260925 14:49:27.652148 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46769
I20260925 14:49:27.652235 7603 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.652349 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41305
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46769
--webserver_port=39489
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:49:27.652413 7603 heartbeater.cc:511] Master 127.0.98.62:46769 requested a full tablet report, sending...
I20260925 14:49:27.652792 6493 ts_manager.cc:194] Re-registered known tserver with Master: 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:27.652905 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35907
I20260925 14:49:27.653002 7602 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.653113 7602 heartbeater.cc:511] Master 127.0.98.60:35907 requested a full tablet report, sending...
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:27.653319 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37941
I20260925 14:49:27.653436 7286 ts_manager.cc:194] Registered new tserver with Master: 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:27.656450 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.639559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54745 (local address 127.0.98.61:45995)
0925 14:49:27.639681 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.639683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.639955 (+ 272us) server_negotiation.cc:407] Connection header received
0925 14:49:27.640996 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.640997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.641010 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.641044 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.641771 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.642557 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.643692 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.643964 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.644015 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.653643 (+ 9628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.653644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.653645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.653653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.655296 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.655655 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.655656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.655656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.655713 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.655910 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.655912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.655912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.656035 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.656110 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.656272 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.656354 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 14:49:27.656488 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.639520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.98.2:54745)
0925 14:49:27.639883 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.639892 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.640869 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.641136 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.641139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.641479 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.641660 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.641663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.642651 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.642652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.643582 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.643584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.644046 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.644047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.644180 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.644867 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.644878 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.653538 (+ 8660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.655378 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.655379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.655379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.655588 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.655778 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.655779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.655780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.655835 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.656103 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.656104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.656157 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.656247 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.656355 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":30,"thread_start_us":97,"threads_started":1}
I20260925 14:49:27.656848 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45995
I20260925 14:49:27.656894 7604 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.656965 7604 heartbeater.cc:511] Master 127.0.98.61:45995 requested a full tablet report, sending...
I20260925 14:49:27.657248 7092 ts_manager.cc:194] Registered new tserver with Master: 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
W20260925 14:49:27.740087 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:27.740440 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:27.740487 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:27.742159 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:27.742219 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:27.742260 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:27.744282 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39489
--tserver_master_addrs=127.0.98.62:46769,127.0.98.61:45995,127.0.98.60:35907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:27.744567 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:27.744856 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:27.750932 7609 server_base.cc:1034] running on GCE node
W20260925 14:49:27.750932 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.751034 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:27.751144 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:27.751472 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:27.751817 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:27.752993 7609 hybrid_clock.cc:648] HybridClock initialized: now 1790347767752946 us; error 65 us; skew 500 ppm
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:27.755707 7609 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:49:27.756260 7609 webserver.cc:460] Webserver started at http://127.0.98.3:39489/ using document root <none> and password file <none>
I20260925 14:49:27.756507 7609 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:27.756592 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:27.758257 7609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:27.758978 7624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.759157 7609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.759253 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "6b61378a78c244f6ad71c8477c0c2acb"
format_stamp: "Formatted at 2026-09-25 14:49:26 on dist-test-slave-z2s5"
server_key: "9d3984c5f54d5e0d178497e08241b047"
server_key_iv: "2a0d29920500a8350283e3d613f04067"
server_key_version: "encryptionkey@0"
I20260925 14:49:27.759845 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:27.763129 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.759701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44962 (local address 127.0.98.60:35907)
0925 14:49:27.759750 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.759752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.759761 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:27.759788 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.759789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.759810 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.759900 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.760095 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.760921 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.762280 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.762780 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.762847 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.762915 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.763041 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.763080 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:27.763141 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.759590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.0.1:44962)
0925 14:49:27.759647 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.759659 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.759709 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.759960 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.759962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.759964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.760016 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.760019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.761016 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.761017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.762162 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.762166 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.762876 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.762877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.762948 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.762952 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.763016 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.763067 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:49:27.763439 7296 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 14:49:27.763579 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1be7adafed984ff798ec3ebe6345eb3e. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.763717 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: LEARNER
I20260925 14:49:27.770462 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:27.772301 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:27.772421 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:27.772596 7609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:27.772894 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:27.772929 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.772951 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:27.773000 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:27.779260 7609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41305
I20260925 14:49:27.779304 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41305 every 8 connection(s)
I20260925 14:49:27.779748 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 25 14:49:27 dist-test-slave-z2s5 krb5kdc[6449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347767, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260925 14:49:27.788340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 7609
I20260925 14:49:27.792356 6534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.780832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55333 (local address 127.0.98.62:46769)
0925 14:49:27.780892 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.780895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.780955 (+ 60us) server_negotiation.cc:407] Connection header received
0925 14:49:27.781965 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.781966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.781978 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.782036 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.783011 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.783523 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.784618 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.784819 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.784905 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.787681 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.787682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.787682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.787689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.790688 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791213 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.791215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.791215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.791275 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791569 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.791570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.791571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.791708 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.791753 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.792162 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.792242 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:27.792891 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.780490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.3:55333)
0925 14:49:27.780890 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.780902 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.781823 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.782119 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.782124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.782655 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.782894 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.782899 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.783652 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.783653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.784515 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.784517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.784913 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.784914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.785041 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.785596 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.785607 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.787562 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.790781 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.790787 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.790790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.791124 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.791351 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791351 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.791351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.791478 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.791801 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.791807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.792000 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.792452 (+ 452us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.792638 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":22,"spinlock_wait_cycles":8064,"thread_start_us":129,"threads_started":1}
I20260925 14:49:27.793157 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.780758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37051 (local address 127.0.98.61:45995)
0925 14:49:27.780805 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.780807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.781002 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:49:27.781945 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.781947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.781960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.781994 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.783022 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.783787 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.784934 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.785220 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.785256 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.788706 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.788708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.788708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.788717 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.791096 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791928 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.791929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.791937 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.791988 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.792309 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.792310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.792311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.792429 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.792487 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.793024 (+ 537us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.793098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:27.793502 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.780781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47805 (local address 127.0.98.60:35907)
0925 14:49:27.780893 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.780895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.780905 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:27.781946 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.781947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.781964 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.781995 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.783009 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.783777 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.784923 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.785210 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.785251 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.788138 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.788139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.788140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.788148 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.791183 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791804 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.791806 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.791806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.791858 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.792535 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.792536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.792537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.792661 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.792702 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.793349 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.793419 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 14:49:27.793532 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.780481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.98.3:47805)
0925 14:49:27.780682 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.780727 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.781816 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.782120 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.782124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.782646 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.782895 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.782901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.783879 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.783880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.784818 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.784821 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.785328 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.785329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.785407 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.786017 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.786022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.788042 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.791257 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791257 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.791258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.791463 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.792059 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.792059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.792059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.792119 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.792875 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.792876 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.792908 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.793319 (+ 411us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.793419 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":34,"thread_start_us":101,"threads_started":1}
I20260925 14:49:27.793771 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.780481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.98.3:37051)
0925 14:49:27.780951 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.780961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.781817 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.782121 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.782124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.782648 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:27.782895 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.782901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.783902 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.783902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.784849 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.784851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.785329 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.785329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.785390 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.785930 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:27.785934 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:27.788613 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:27.791533 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.791534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.791534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.791717 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.792192 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:27.792192 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:27.792193 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:27.792242 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:27.792749 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:27.792750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:27.792787 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.793601 (+ 814us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.793698 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":84,"thread_start_us":98,"threads_started":1}
I20260925 14:49:27.794112 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.61:45995
I20260925 14:49:27.794200 7740 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.794281 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.60:35907
I20260925 14:49:27.794351 7738 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.794435 7740 heartbeater.cc:511] Master 127.0.98.61:45995 requested a full tablet report, sending...
I20260925 14:49:27.794448 7738 heartbeater.cc:511] Master 127.0.98.60:35907 requested a full tablet report, sending...
I20260925 14:49:27.794636 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46769
I20260925 14:49:27.794682 7739 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:27.794744 7739 heartbeater.cc:511] Master 127.0.98.62:46769 requested a full tablet report, sending...
I20260925 14:49:27.794905 7092 ts_manager.cc:194] Registered new tserver with Master: 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305)
I20260925 14:49:27.794981 7286 ts_manager.cc:194] Registered new tserver with Master: 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305)
I20260925 14:49:27.795127 6493 ts_manager.cc:194] Re-registered known tserver with Master: 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305)
I20260925 14:49:27.795681 6493 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55333
I20260925 14:49:27.801512 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.790189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.0.98.61:45995)
0925 14:49:27.790300 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.790302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.790583 (+ 281us) server_negotiation.cc:407] Connection header received
0925 14:49:27.790687 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.790688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.790701 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.790744 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.794529 (+ 3785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.795267 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.796353 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.796619 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.796697 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.798599 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.798600 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.798601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.798607 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.800368 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.800751 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.800752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.800753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.800812 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.801008 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.801009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.801009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.801147 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.801239 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.801332 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.801401 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260925 14:49:27.802230 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.789815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.98.62:46769)
0925 14:49:27.789968 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.789970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.790480 (+ 510us) server_negotiation.cc:407] Connection header received
0925 14:49:27.791162 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.791164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.791179 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.791233 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.792418 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.793245 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.796400 (+ 3155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.796743 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.796784 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.798650 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.798652 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.798652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.798660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.800692 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.801143 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.801144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.801144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.801207 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.801532 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.801533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.801534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.801678 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.801768 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.802032 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.802129 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260925 14:49:27.802253 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.789550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44974 (local address 127.0.98.60:35907)
0925 14:49:27.789664 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.789666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.790470 (+ 804us) server_negotiation.cc:407] Connection header received
0925 14:49:27.790594 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.790595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.790624 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.790669 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:27.791651 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.792556 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.796985 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.797303 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.797364 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.799085 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:27.799086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:27.799087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:27.799094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:27.800870 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.801301 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.801302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.801302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.801355 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:27.801586 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:27.801587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:27.801587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:27.801741 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:27.801837 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.802112 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.802177 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
I20260925 14:49:27.804177 6493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:27.805859 6493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:27.806643 7054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: true } } }
I20260925 14:49:27.806738 7054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.807700 7053 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: attempting to promote NON_VOTER e4b1e17e0c4b417bb5730eb1641238aa to VOTER
I20260925 14:49:27.807835 7053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } }
I20260925 14:49:27.808254 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 LEARNER]: Refusing update from remote peer 1be7adafed984ff798ec3ebe6345eb3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 14:49:27.808310 7102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Refusing update from remote peer 1be7adafed984ff798ec3ebe6345eb3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 14:49:27.808537 7239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 14:49:27.808625 7054 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 14:49:27.809504 7296 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.809710 7102 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.810328 7247 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.821260 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.821377 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.821445 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.821491 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.821523 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.821568 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:27.822125 7246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.822229 7246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:49:27.822299 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1be7adafed984ff798ec3ebe6345eb3e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:27.822414 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:49:27.827221 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.822509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35853 (local address 127.0.0.1:59124)
0925 14:49:27.822550 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.822565 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.822633 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.823488 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.823489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.823491 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.823553 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.823555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.824989 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.824990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.825967 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.825969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.826970 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.826971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.827034 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.827038 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.827101 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.827142 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 14:49:27.827237 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.822662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.0.98.2:35853)
0925 14:49:27.823085 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.823090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.823101 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:27.823143 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.823146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.823163 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.823632 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.823694 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.824886 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.826069 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.826883 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.826951 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.827043 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.827117 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.827156 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":95,"threads_started":1}
I20260925 14:49:27.827980 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.822697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41367 (local address 127.0.0.1:54480)
0925 14:49:27.822910 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.822923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.823490 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.824047 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.824048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.824050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.824107 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.824109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.825557 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.825558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.826662 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.826665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.827687 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.827688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.827771 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.827775 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.827818 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.827874 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":86,"threads_started":1}
I20260925 14:49:27.827986 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.822878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41305 (local address 127.0.0.1:50540)
0925 14:49:27.823792 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.823803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.823846 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.824155 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.824156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.824158 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.824187 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.824188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.825851 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.825852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.826685 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.826687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.827757 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.827757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.827813 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.827817 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.827845 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.827877 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"thread_start_us":65,"threads_started":1}
I20260925 14:49:27.828179 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.822900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54480 (local address 127.0.98.1:41367)
0925 14:49:27.823854 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.823861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.823871 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:27.823913 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.823915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.823934 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.824185 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.824248 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.825370 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.826749 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.827589 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.827677 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.828018 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.828048 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.828091 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":917,"thread_start_us":69,"threads_started":1}
I20260925 14:49:27.828627 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.823598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50540 (local address 127.0.98.3:41305)
0925 14:49:27.823999 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.824003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.824012 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:27.824048 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.824049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.824064 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.824253 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.824305 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.825776 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.826784 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.827660 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.828396 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.828476 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.828503 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.828548 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":79,"threads_started":1}
I20260925 14:49:27.828562 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 657d2bf335a94449b042dcb23ec682b7 (DEFAULT_TABLE table=test-table [id=919fd3b4a46c48bd8e657572071d1dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:27.828830 7399 tablet_service.cc:1467] Processing CreateTablet for tablet 657d2bf335a94449b042dcb23ec682b7 (DEFAULT_TABLE table=test-table [id=919fd3b4a46c48bd8e657572071d1dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:27.828956 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657d2bf335a94449b042dcb23ec682b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:27.829166 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657d2bf335a94449b042dcb23ec682b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:27.829227 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 657d2bf335a94449b042dcb23ec682b7 (DEFAULT_TABLE table=test-table [id=919fd3b4a46c48bd8e657572071d1dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:27.829519 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 657d2bf335a94449b042dcb23ec682b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:27.831980 7767 tablet_bootstrap.cc:492] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Bootstrap starting.
I20260925 14:49:27.832587 7767 tablet_bootstrap.cc:654] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:27.832865 7768 tablet_bootstrap.cc:492] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Bootstrap starting.
I20260925 14:49:27.832998 7769 tablet_bootstrap.cc:492] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Bootstrap starting.
I20260925 14:49:27.833055 7767 log.cc:824] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:27.833475 7768 tablet_bootstrap.cc:654] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:27.833618 7769 tablet_bootstrap.cc:654] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:27.833794 7768 log.cc:824] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:27.833905 7767 tablet_bootstrap.cc:492] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: No bootstrap required, opened a new log
I20260925 14:49:27.833933 7769 log.cc:824] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:27.833976 7767 ts_tablet_manager.cc:1397] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:27.834553 7768 tablet_bootstrap.cc:492] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: No bootstrap required, opened a new log
I20260925 14:49:27.834623 7768 ts_tablet_manager.cc:1397] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:27.834715 7769 tablet_bootstrap.cc:492] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: No bootstrap required, opened a new log
I20260925 14:49:27.834781 7769 ts_tablet_manager.cc:1397] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:27.836095 7769 raft_consensus.cc:348] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.836266 7769 raft_consensus.cc:374] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:27.836313 7769 raft_consensus.cc:729] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b61378a78c244f6ad71c8477c0c2acb, State: Initialized, Role: FOLLOWER
I20260925 14:49:27.836207 7767 raft_consensus.cc:348] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.836364 7767 raft_consensus.cc:374] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:27.836402 7767 raft_consensus.cc:729] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c36349d6824cedb50b403f3e96638f, State: Initialized, Role: FOLLOWER
I20260925 14:49:27.836467 7769 consensus_queue.cc:260] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.836519 7767 consensus_queue.cc:260] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.836768 7767 ts_tablet_manager.cc:1428] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:49:27.836989 7769 ts_tablet_manager.cc:1428] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:49:27.837620 7739 heartbeater.cc:503] Master 127.0.98.62:46769 was elected leader, sending a full tablet report...
I20260925 14:49:27.837322 7603 heartbeater.cc:503] Master 127.0.98.62:46769 was elected leader, sending a full tablet report...
I20260925 14:49:27.838275 7768 raft_consensus.cc:348] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.838704 7768 raft_consensus.cc:374] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:27.838762 7768 raft_consensus.cc:729] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfad53226de440e8e44f810d606b84d, State: Initialized, Role: FOLLOWER
I20260925 14:49:27.838923 7768 consensus_queue.cc:260] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.839226 7768 ts_tablet_manager.cc:1428] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 14:49:27.839304 7466 heartbeater.cc:503] Master 127.0.98.62:46769 was elected leader, sending a full tablet report...
W20260925 14:49:27.890336 7605 tablet.cc:2369] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:27.967511 7775 raft_consensus.cc:482] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:27.967648 7775 raft_consensus.cc:504] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:27.968087 7775 leader_election.cc:288] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305), 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:27.971153 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.968237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41305 (local address 127.0.98.1:52247)
0925 14:49:27.968289 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.968303 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.968347 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.968527 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.968528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.968530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.968826 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.968828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.969474 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.969475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.970200 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.970201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.970907 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.970908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.970968 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.970972 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.971017 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.971049 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:49:27.971158 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.968351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52247 (local address 127.0.98.3:41305)
0925 14:49:27.968402 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.968404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.968411 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:27.968443 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.968444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.968453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.968522 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.968906 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.969376 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.970293 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.970830 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.970903 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.970914 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.971041 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.971078 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:49:27.971386 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.968257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35853 (local address 127.0.98.1:43443)
0925 14:49:27.968306 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:27.968318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:27.968358 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:27.968537 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:27.968538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:27.968540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:27.968629 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.968630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.969704 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.969704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.970428 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:27.970430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.971136 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.971137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:27.971210 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.971212 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:27.971268 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:49:27.971309 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 14:49:27.971518 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:27.968367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43443 (local address 127.0.98.2:35853)
0925 14:49:27.968415 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:27.968417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:27.968424 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:27.968447 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:27.968448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:27.968458 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:27.968535 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:27.968700 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.969392 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.970521 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:27.971071 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:27.971138 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:27.971147 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:27.971395 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 14:49:27.971436 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:49:27.971731 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d2bf335a94449b042dcb23ec682b7" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b61378a78c244f6ad71c8477c0c2acb" is_pre_election: true
I20260925 14:49:27.971807 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d2bf335a94449b042dcb23ec682b7" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c36349d6824cedb50b403f3e96638f" is_pre_election: true
I20260925 14:49:27.971902 7692 raft_consensus.cc:2454] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 0.
I20260925 14:49:27.971918 7556 raft_consensus.cc:2454] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 0.
I20260925 14:49:27.972138 7352 leader_election.cc:302] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c36349d6824cedb50b403f3e96638f, fdfad53226de440e8e44f810d606b84d; no voters:
I20260925 14:49:27.972312 7775 raft_consensus.cc:2781] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:27.972350 7775 raft_consensus.cc:482] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:27.972365 7775 raft_consensus.cc:3037] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Advancing to term 1
W20260925 14:49:27.984927 7468 tablet.cc:2369] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:28.015365 7775 raft_consensus.cc:504] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:28.015687 7775 leader_election.cc:288] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 election: Requested vote from peers 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305), 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:28.015815 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d2bf335a94449b042dcb23ec682b7" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c36349d6824cedb50b403f3e96638f"
I20260925 14:49:28.015825 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "657d2bf335a94449b042dcb23ec682b7" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b61378a78c244f6ad71c8477c0c2acb"
I20260925 14:49:28.015913 7692 raft_consensus.cc:3037] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:28.015913 7556 raft_consensus.cc:3037] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:28.016904 7556 raft_consensus.cc:2454] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 1.
I20260925 14:49:28.017122 7352 leader_election.cc:302] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c36349d6824cedb50b403f3e96638f, fdfad53226de440e8e44f810d606b84d; no voters:
I20260925 14:49:28.017246 7775 raft_consensus.cc:2781] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:28.017272 7692 raft_consensus.cc:2454] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 1.
I20260925 14:49:28.017379 7775 raft_consensus.cc:686] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 1 LEADER]: Becoming Leader. State: Replica: fdfad53226de440e8e44f810d606b84d, State: Running, Role: LEADER
I20260925 14:49:28.017498 7775 consensus_queue.cc:237] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:28.018524 6494 catalog_manager.cc:5495] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfad53226de440e8e44f810d606b84d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fdfad53226de440e8e44f810d606b84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } health_report { overall_health: UNKNOWN } } }
W20260925 14:49:28.030573 7741 tablet.cc:2369] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:28.053771 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:28.051319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.0.98.1:41367)
0925 14:49:28.051364 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:28.051366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:28.051394 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:49:28.051440 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:28.051452 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:28.051472 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:28.051590 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:28.051771 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.052284 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.053118 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.053297 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.053351 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:28.053507 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:28.053605 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:28.053665 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:28.053699 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:49:28.053714 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:49:28.055753 7556 raft_consensus.cc:1261] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Refusing update from remote peer fdfad53226de440e8e44f810d606b84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:28.055761 7692 raft_consensus.cc:1261] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Refusing update from remote peer fdfad53226de440e8e44f810d606b84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:28.056140 7776 consensus_queue.cc:1035] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:28.056267 7775 consensus_queue.cc:1035] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:28.060418 7781 mvcc.cc:205] Tried to move back new op lower bound from 7333264457953341440 to 7333264457800871936. Current Snapshot: MvccSnapshot[applied={T|T < 7333264457953341440}]
I20260925 14:49:28.061992 6493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:28.062083 6493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:28.063441 6493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1be7adafed984ff798ec3ebe6345eb3e: Sending DeleteTablet for 3 replicas of tablet 657d2bf335a94449b042dcb23ec682b7
I20260925 14:49:28.063879 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 657d2bf335a94449b042dcb23ec682b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59124
I20260925 14:49:28.064141 7789 tablet_replica.cc:330] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: stopping tablet replica
I20260925 14:49:28.064250 7789 raft_consensus.cc:2229] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:28.064263 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet 657d2bf335a94449b042dcb23ec682b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54480
I20260925 14:49:28.064327 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 657d2bf335a94449b042dcb23ec682b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:28 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50540
I20260925 14:49:28.064374 7789 raft_consensus.cc:2258] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:28.064832 7789 ts_tablet_manager.cc:1905] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:28.064883 7790 tablet_replica.cc:330] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: stopping tablet replica
I20260925 14:49:28.064965 7790 raft_consensus.cc:2229] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:28.065079 7790 raft_consensus.cc:2258] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:28.065487 7790 ts_tablet_manager.cc:1905] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:28.064558 7791 tablet_replica.cc:330] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: stopping tablet replica
I20260925 14:49:28.065896 7791 raft_consensus.cc:2229] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:28.065961 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6462
I20260925 14:49:28.065995 7791 raft_consensus.cc:2258] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:28.067142 7791 ts_tablet_manager.cc:1905] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:28.075703 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:28.065572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.98.60:35907)
0925 14:49:28.065616 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:28.065619 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:28.066983 (+ 1364us) server_negotiation.cc:407] Connection header received
0925 14:49:28.067088 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:28.067089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:28.067103 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:28.067133 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:28.067930 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.068512 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.069217 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.069441 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.069531 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:28.071325 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:28.071326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:28.071326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:28.071335 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:28.072926 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:28.073335 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:28.073336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:28.073336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:28.073405 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:28.073635 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:28.073636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:28.073637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:28.073794 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:28.073879 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:28.075507 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0925 14:49:28.075607 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:28.083148 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:28.075509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.0.98.60:35907)
0925 14:49:28.075692 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:28.075694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:28.075701 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:28.075721 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:28.075722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:28.075732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:28.075784 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:28.076933 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.077466 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.078418 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.078618 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.078693 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:28.080089 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:28.080090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:28.080091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:28.080097 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:28.081829 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:28.082308 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:28.082309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:28.082309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:28.082378 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:28.082606 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:28.082607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:28.082607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:28.082745 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:28.082853 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:28.082942 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 14:49:28.083020 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260925 14:49:28.083536 7753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:28.075228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60676 (local address 127.0.98.61:45995)
0925 14:49:28.075273 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:28.075275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:28.075541 (+ 266us) server_negotiation.cc:407] Connection header received
0925 14:49:28.075571 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:28.075572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:28.075586 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:28.075638 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:28.076506 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.077291 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.078317 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:28.078660 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:28.078729 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:28.080388 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:28.080390 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:28.080390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:28.080398 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:28.081996 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:28.082476 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:28.082478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:28.082478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:28.082541 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:28.082756 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:28.082757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:28.082758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:28.082888 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:28.082974 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:28.083157 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 14:49:28.083476 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:49:28.117878 7789 ts_tablet_manager.cc:1918] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:28.117879 7790 ts_tablet_manager.cc:1918] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:28.117986 7790 log.cc:1196] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/657d2bf335a94449b042dcb23ec682b7
I20260925 14:49:28.117986 7789 log.cc:1196] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/657d2bf335a94449b042dcb23ec682b7
I20260925 14:49:28.118268 7790 ts_tablet_manager.cc:1939] T 657d2bf335a94449b042dcb23ec682b7 P fdfad53226de440e8e44f810d606b84d: Deleting consensus metadata
I20260925 14:49:28.118283 7789 ts_tablet_manager.cc:1939] T 657d2bf335a94449b042dcb23ec682b7 P 43c36349d6824cedb50b403f3e96638f: Deleting consensus metadata
W20260925 14:49:28.139490 7353 connection.cc:441] server connection from 127.0.0.1:54480 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54480 (request call id 1) could send its response
W20260925 14:49:28.139498 7489 connection.cc:441] server connection from 127.0.0.1:59124 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59124 (request call id 1) could send its response
I20260925 14:49:28.169934 7791 ts_tablet_manager.cc:1918] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:28.170042 7791 log.cc:1196] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/657d2bf335a94449b042dcb23ec682b7
I20260925 14:49:28.170351 7791 ts_tablet_manager.cc:1939] T 657d2bf335a94449b042dcb23ec682b7 P 6b61378a78c244f6ad71c8477c0c2acb: Deleting consensus metadata
W20260925 14:49:28.172391 7626 connection.cc:441] server connection from 127.0.0.1:50540 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50540 (request call id 1) could send its response
I20260925 14:49:29.061163 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.060882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:32903)
0925 14:49:29.061030 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.061106 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":53,"threads_started":1}
I20260925 14:49:29.061324 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.060755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.3:60199)
0925 14:49:29.061153 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.061270 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":67,"threads_started":1}
I20260925 14:49:29.061368 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.061026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:46997)
0925 14:49:29.061188 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.061327 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":52,"threads_started":1}
W20260925 14:49:29.061383 7466 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
W20260925 14:49:29.061497 7739 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
W20260925 14:49:29.061654 7603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46769 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
I20260925 14:49:29.563953 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1be7adafed984ff798ec3ebe6345eb3e)
I20260925 14:49:29.564141 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1be7adafed984ff798ec3ebe6345eb3e)
I20260925 14:49:29.564093 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } }
I20260925 14:49:29.564235 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } }
I20260925 14:49:29.564433 7803 leader_election.cc:288] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769), 242f729351b44efa9652bb5f9f867b0d (127.0.98.61:45995)
I20260925 14:49:29.564492 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769), e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60:35907)
I20260925 14:49:29.564900 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.564596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49536)
0925 14:49:29.564797 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.564851 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":48,"threads_started":1}
I20260925 14:49:29.564917 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.564618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49548)
0925 14:49:29.564725 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.564866 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":50,"threads_started":1}
W20260925 14:49:29.565052 7077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
W20260925 14:49:29.565127 7271 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
I20260925 14:49:29.565482 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.565405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49552)
0925 14:49:29.565437 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.565458 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 14:49:29.565560 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
I20260925 14:49:29.566198 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.566082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49560)
0925 14:49:29.566139 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.566167 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 14:49:29.566322 7271 leader_election.cc:334] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
I20260925 14:49:29.567896 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.564759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.0.98.60:35907)
0925 14:49:29.565024 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:29.565026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:29.565033 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:29.565056 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:29.565057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:29.565067 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:29.565129 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:29.565307 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.566012 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.567094 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.567587 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.567642 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.567651 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:29.567790 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 14:49:29.567820 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":40,"threads_started":1}
I20260925 14:49:29.567898 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.564769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35907 (local address 127.0.0.1:45000)
0925 14:49:29.564857 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.564868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:29.564925 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:29.565176 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:29.565177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:29.565179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:29.565228 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.565231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.566100 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.566101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.567010 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.567013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.567651 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.567652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.567712 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.567716 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:29.567772 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:49:29.567810 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":44,"threads_started":1}
I20260925 14:49:29.568135 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.564630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45995 (local address 127.0.0.1:60678)
0925 14:49:29.564863 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.564873 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:29.564929 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:29.565172 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:29.565173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:29.565175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:29.565231 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.565234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.566284 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.566285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.567265 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.567267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.567891 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.567892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.567957 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.567960 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:29.568021 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:49:29.568068 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":40,"threads_started":1}
I20260925 14:49:29.568236 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "242f729351b44efa9652bb5f9f867b0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" is_pre_election: true
I20260925 14:49:29.568336 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 242f729351b44efa9652bb5f9f867b0d in term 1.
I20260925 14:49:29.568393 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.564749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.98.61:45995)
0925 14:49:29.565012 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:29.565014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:29.565021 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:29.565046 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:29.565046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:29.565056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:29.565122 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:29.565312 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.566195 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.567344 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.567819 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.567880 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.567888 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:29.568274 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 14:49:29.568315 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":55,"threads_started":1}
I20260925 14:49:29.568481 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 242f729351b44efa9652bb5f9f867b0d, e4b1e17e0c4b417bb5730eb1641238aa; no voters: 1be7adafed984ff798ec3ebe6345eb3e
I20260925 14:49:29.568599 7804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:49:29.568568 7102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "242f729351b44efa9652bb5f9f867b0d" is_pre_election: true
I20260925 14:49:29.568650 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1be7adafed984ff798ec3ebe6345eb3e)
I20260925 14:49:29.568681 7804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:49:29.569344 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } }
I20260925 14:49:29.569473 7102 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4b1e17e0c4b417bb5730eb1641238aa in current term 2: Already voted for candidate 242f729351b44efa9652bb5f9f867b0d in this term.
I20260925 14:49:29.569507 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 2 election: Requested vote from peers 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769), e4b1e17e0c4b417bb5730eb1641238aa (127.0.98.60:35907)
I20260925 14:49:29.569662 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.569574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49562)
0925 14:49:29.569612 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.569642 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:49:29.569643 7296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "242f729351b44efa9652bb5f9f867b0d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4b1e17e0c4b417bb5730eb1641238aa"
I20260925 14:49:29.569690 7272 leader_election.cc:302] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4b1e17e0c4b417bb5730eb1641238aa; no voters: 1be7adafed984ff798ec3ebe6345eb3e, 242f729351b44efa9652bb5f9f867b0d
I20260925 14:49:29.569694 7296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:49:29.570003 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.569870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49564)
0925 14:49:29.569923 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.569978 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:49:29.570075 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769): Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
I20260925 14:49:29.570549 7296 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 242f729351b44efa9652bb5f9f867b0d in term 2.
I20260925 14:49:29.570621 7803 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:49:29.570719 7076 leader_election.cc:302] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 242f729351b44efa9652bb5f9f867b0d, e4b1e17e0c4b417bb5730eb1641238aa; no voters: 1be7adafed984ff798ec3ebe6345eb3e
I20260925 14:49:29.570807 7804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:49:29.570941 7804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [term 2 LEADER]: Becoming Leader. State: Replica: 242f729351b44efa9652bb5f9f867b0d, State: Running, Role: LEADER
I20260925 14:49:29.571040 7804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } }
I20260925 14:49:29.571376 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 242f729351b44efa9652bb5f9f867b0d. Latest consensus state: current_term: 2 leader_uuid: "242f729351b44efa9652bb5f9f867b0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:29.571473 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:29.571641 7815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:29.572016 7815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:29.572069 7815 catalog_manager.cc:1259] Loaded cluster ID: 85eb772cae084a0ba072ec26be539ce2
I20260925 14:49:29.572085 7815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:29.572199 7815 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:29.572249 7815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Loaded TSK: 0
I20260925 14:49:29.572338 7815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:29.624907 7091 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:29.626591 7091 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:29.627615 7296 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [term 2 FOLLOWER]: Refusing update from remote peer 242f729351b44efa9652bb5f9f867b0d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 14:49:29.627640 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.627480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49584)
0925 14:49:29.627556 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.627594 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:49:29.627856 7813 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 14:49:29.627990 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.627878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.0.1:49586)
0925 14:49:29.627922 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.627956 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 14:49:29.628069 7077 consensus_peers.cc:487] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d -> Peer 1be7adafed984ff798ec3ebe6345eb3e (127.0.98.62:46769): Couldn't send request to peer 1be7adafed984ff798ec3ebe6345eb3e. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:49:29.628470 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 242f729351b44efa9652bb5f9f867b0d. Latest consensus state: current_term: 2 leader_uuid: "242f729351b44efa9652bb5f9f867b0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:29.628577 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:49:29.628988 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "242f729351b44efa9652bb5f9f867b0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:29.629072 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:29.629148 7803 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "242f729351b44efa9652bb5f9f867b0d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "1be7adafed984ff798ec3ebe6345eb3e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46769 } } peers { permanent_uuid: "242f729351b44efa9652bb5f9f867b0d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45995 } attrs { promote: false } } peers { permanent_uuid: "e4b1e17e0c4b417bb5730eb1641238aa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35907 } attrs { promote: false } } }
I20260925 14:49:29.629238 7803 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4b1e17e0c4b417bb5730eb1641238aa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:49:29.633934 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.630771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41305 (local address 127.0.0.1:50542)
0925 14:49:29.630846 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.630858 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:29.630917 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:29.631229 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:29.631230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:29.631233 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:29.631296 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.631299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.632305 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.632306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.633042 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.633044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.633696 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.633696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.633761 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.633764 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:29.633823 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:49:29.633859 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":2}
I20260925 14:49:29.633953 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.630920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.0.98.3:41305)
0925 14:49:29.631032 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:29.631034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:29.631042 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:29.631068 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:29.631068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:29.631080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:29.631363 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:29.631399 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.632196 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.633160 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.633613 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.633685 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.633697 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:29.633837 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 14:49:29.633876 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260925 14:49:29.634289 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.630771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35853 (local address 127.0.0.1:59130)
0925 14:49:29.630960 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.630969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:29.631008 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:29.631312 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:29.631312 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:29.631314 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:29.631359 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.631360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.632025 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.632026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.633138 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.633140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.634059 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.634060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.634118 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.634123 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:29.634172 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:49:29.634208 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":20}
I20260925 14:49:29.634410 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.630861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41367 (local address 127.0.0.1:54494)
0925 14:49:29.631034 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.631048 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:29.631081 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:29.631373 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:29.631373 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:29.631375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:29.631408 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.631409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.632279 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.632280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.633370 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:29.633372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.634161 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.634161 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:29.634211 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.634213 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:29.634269 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:49:29.634303 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":21,"thread_start_us":65,"threads_started":1}
I20260925 14:49:29.634440 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.631137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.98.1:41367)
0925 14:49:29.631243 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:29.631244 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:29.631251 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:29.631274 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:29.631275 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:29.631285 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:29.631482 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:29.631519 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.632204 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.633470 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.634074 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.634160 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.634173 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:29.634277 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 14:49:29.634330 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260925 14:49:29.634584 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.630866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.0.98.2:35853)
0925 14:49:29.631177 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:29.631179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:29.631187 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:29.631213 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:29.631214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:29.631224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:29.631424 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:29.631462 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.631934 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.633442 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.633951 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.634059 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.634073 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:29.634473 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 14:49:29.634527 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":51,"threads_started":1}
I20260925 14:49:29.634804 7535 tablet_service.cc:1467] Processing CreateTablet for tablet e89d628905a74300a9df95b88c81fbcf (DEFAULT_TABLE table=test-table [id=122a21e1906945daa98396fc95361b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:29.634812 7399 tablet_service.cc:1467] Processing CreateTablet for tablet e89d628905a74300a9df95b88c81fbcf (DEFAULT_TABLE table=test-table [id=122a21e1906945daa98396fc95361b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:29.634927 7399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89d628905a74300a9df95b88c81fbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:29.634938 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89d628905a74300a9df95b88c81fbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:29.635100 7672 tablet_service.cc:1467] Processing CreateTablet for tablet e89d628905a74300a9df95b88c81fbcf (DEFAULT_TABLE table=test-table [id=122a21e1906945daa98396fc95361b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:29.635195 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e89d628905a74300a9df95b88c81fbcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:29.635949 7826 tablet_bootstrap.cc:492] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Bootstrap starting.
I20260925 14:49:29.636359 7826 tablet_bootstrap.cc:654] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:29.636409 7827 tablet_bootstrap.cc:492] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d: Bootstrap starting.
I20260925 14:49:29.636708 7828 tablet_bootstrap.cc:492] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Bootstrap starting.
I20260925 14:49:29.637058 7828 tablet_bootstrap.cc:654] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:29.637068 7827 tablet_bootstrap.cc:654] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:29.637122 7826 tablet_bootstrap.cc:492] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: No bootstrap required, opened a new log
I20260925 14:49:29.637178 7826 ts_tablet_manager.cc:1397] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:29.637315 7826 raft_consensus.cc:348] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.637395 7826 raft_consensus.cc:374] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:29.637420 7826 raft_consensus.cc:729] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c36349d6824cedb50b403f3e96638f, State: Initialized, Role: FOLLOWER
I20260925 14:49:29.637470 7826 consensus_queue.cc:260] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.637576 7828 tablet_bootstrap.cc:492] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: No bootstrap required, opened a new log
I20260925 14:49:29.637627 7828 ts_tablet_manager.cc:1397] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:29.637635 7826 ts_tablet_manager.cc:1428] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:29.637756 7828 raft_consensus.cc:348] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.637825 7828 raft_consensus.cc:374] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:29.637856 7828 raft_consensus.cc:729] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b61378a78c244f6ad71c8477c0c2acb, State: Initialized, Role: FOLLOWER
I20260925 14:49:29.637910 7828 consensus_queue.cc:260] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.638052 7828 ts_tablet_manager.cc:1428] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:29.638469 7604 heartbeater.cc:503] Master 127.0.98.61:45995 was elected leader, sending a full tablet report...
I20260925 14:49:29.639038 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.638227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:59901)
0925 14:49:29.638949 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.639003 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":93,"threads_started":1}
I20260925 14:49:29.639117 7827 tablet_bootstrap.cc:492] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d: No bootstrap required, opened a new log
I20260925 14:49:29.639165 7827 ts_tablet_manager.cc:1397] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:49:29.639274 7827 raft_consensus.cc:348] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.639336 7827 raft_consensus.cc:374] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:29.639354 7827 raft_consensus.cc:729] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdfad53226de440e8e44f810d606b84d, State: Initialized, Role: FOLLOWER
I20260925 14:49:29.639393 7827 consensus_queue.cc:260] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.639427 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.639238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:52051)
0925 14:49:29.639374 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.639398 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260925 14:49:29.639542 7827 ts_tablet_manager.cc:1428] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:29.639803 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.638547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.3:56127)
0925 14:49:29.639724 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.639761 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1155,"thread_start_us":70,"threads_started":1}
I20260925 14:49:29.640086 7467 heartbeater.cc:503] Master 127.0.98.61:45995 was elected leader, sending a full tablet report...
I20260925 14:49:29.640198 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.639696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:49609)
0925 14:49:29.640110 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.640160 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":72,"threads_started":1}
I20260925 14:49:29.707072 7837 raft_consensus.cc:482] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:29.707181 7837 raft_consensus.cc:504] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.707405 7837 leader_election.cc:288] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305), 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:29.707615 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89d628905a74300a9df95b88c81fbcf" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c36349d6824cedb50b403f3e96638f" is_pre_election: true
I20260925 14:49:29.707640 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89d628905a74300a9df95b88c81fbcf" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b61378a78c244f6ad71c8477c0c2acb" is_pre_election: true
I20260925 14:49:29.707721 7556 raft_consensus.cc:2454] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 0.
I20260925 14:49:29.707752 7692 raft_consensus.cc:2454] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 0.
I20260925 14:49:29.707943 7352 leader_election.cc:302] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c36349d6824cedb50b403f3e96638f, fdfad53226de440e8e44f810d606b84d; no voters:
I20260925 14:49:29.708056 7837 raft_consensus.cc:2781] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:29.708096 7837 raft_consensus.cc:482] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:29.708122 7837 raft_consensus.cc:3037] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:29.708897 7837 raft_consensus.cc:504] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.709059 7837 leader_election.cc:288] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 election: Requested vote from peers 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305), 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853)
I20260925 14:49:29.709203 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89d628905a74300a9df95b88c81fbcf" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c36349d6824cedb50b403f3e96638f"
I20260925 14:49:29.709229 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e89d628905a74300a9df95b88c81fbcf" candidate_uuid: "fdfad53226de440e8e44f810d606b84d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b61378a78c244f6ad71c8477c0c2acb"
I20260925 14:49:29.709280 7556 raft_consensus.cc:3037] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:29.709295 7692 raft_consensus.cc:3037] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:29.710011 7692 raft_consensus.cc:2454] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 1.
I20260925 14:49:29.710011 7556 raft_consensus.cc:2454] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdfad53226de440e8e44f810d606b84d in term 1.
I20260925 14:49:29.710189 7352 leader_election.cc:302] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c36349d6824cedb50b403f3e96638f, fdfad53226de440e8e44f810d606b84d; no voters:
I20260925 14:49:29.710350 7837 raft_consensus.cc:2781] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:29.710544 7837 raft_consensus.cc:686] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [term 1 LEADER]: Becoming Leader. State: Replica: fdfad53226de440e8e44f810d606b84d, State: Running, Role: LEADER
I20260925 14:49:29.710592 7837 consensus_queue.cc:237] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } }
I20260925 14:49:29.711046 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.710823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:43355)
0925 14:49:29.710892 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.711014 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 14:49:29.711239 7092 catalog_manager.cc:5495] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d reported cstate change: term changed from 0 to 1, leader changed from <none> to fdfad53226de440e8e44f810d606b84d (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fdfad53226de440e8e44f810d606b84d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdfad53226de440e8e44f810d606b84d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:29.727728 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.725590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.1:41367)
0925 14:49:29.725647 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:29.725649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:29.725657 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:29.725707 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:29.725708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:29.725720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:29.725766 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:29.725870 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.726328 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.727132 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:29.727315 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:29.727370 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:29.727538 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:29.727564 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:29.727599 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:29.727650 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:49:29.727670 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:29.728785 7692 raft_consensus.cc:1261] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Refusing update from remote peer fdfad53226de440e8e44f810d606b84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:29.728785 7556 raft_consensus.cc:1261] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Refusing update from remote peer fdfad53226de440e8e44f810d606b84d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:29.729009 7839 consensus_queue.cc:1035] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b61378a78c244f6ad71c8477c0c2acb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:29.729122 7837 consensus_queue.cc:1035] T e89d628905a74300a9df95b88c81fbcf P fdfad53226de440e8e44f810d606b84d [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c36349d6824cedb50b403f3e96638f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:29.729398 7740 heartbeater.cc:503] Master 127.0.98.61:45995 was elected leader, sending a full tablet report...
I20260925 14:49:29.729918 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.729789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:51857)
0925 14:49:29.729830 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.729883 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 14:49:29.730146 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.729871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.3:46701)
0925 14:49:29.730085 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.730113 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260925 14:49:29.730573 7842 mvcc.cc:205] Tried to move back new op lower bound from 7333264464806756352 to 7333264464735293440. Current Snapshot: MvccSnapshot[applied={T|T < 7333264464806756352}]
I20260925 14:49:29.731040 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.730907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:46049)
0925 14:49:29.730983 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.731009 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 14:49:29.731467 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.731220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.3:47837)
0925 14:49:29.731409 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.731437 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260925 14:49:29.732249 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.732116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.1:42103)
0925 14:49:29.732194 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.732220 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 14:49:29.732532 7845 mvcc.cc:205] Tried to move back new op lower bound from 7333264464806756352 to 7333264464735293440. Current Snapshot: MvccSnapshot[applied={T|T < 7333264464806756352}]
I20260925 14:49:29.732735 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:29.732648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46769 (local address 127.0.98.2:39897)
0925 14:49:29.732682 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.732713 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 14:49:29.732998 7091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:29.733093 7091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:29.734306 7091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 242f729351b44efa9652bb5f9f867b0d: Sending DeleteTablet for 3 replicas of tablet e89d628905a74300a9df95b88c81fbcf
I20260925 14:49:29.734709 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet e89d628905a74300a9df95b88c81fbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59130
I20260925 14:49:29.734726 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet e89d628905a74300a9df95b88c81fbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50542
I20260925 14:49:29.734767 7399 tablet_service.cc:1514] Processing DeleteTablet for tablet e89d628905a74300a9df95b88c81fbcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:29 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54494
I20260925 14:49:29.734983 7850 tablet_replica.cc:330] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: stopping tablet replica
I20260925 14:49:29.735036 7850 raft_consensus.cc:2229] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:29.735102 7850 raft_consensus.cc:2258] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:29.735095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 7240
I20260925 14:49:29.735659 7851 tablet_replica.cc:330] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: stopping tablet replica
I20260925 14:49:29.735716 7851 raft_consensus.cc:2229] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:29.735764 7851 raft_consensus.cc:2258] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:29.735816 7850 ts_tablet_manager.cc:1905] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:29.736101 7851 ts_tablet_manager.cc:1905] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:29.737582 7851 ts_tablet_manager.cc:1918] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:29.737624 7851 log.cc:1196] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/e89d628905a74300a9df95b88c81fbcf
I20260925 14:49:29.737713 7850 ts_tablet_manager.cc:1918] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:29.737768 7850 log.cc:1196] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/e89d628905a74300a9df95b88c81fbcf
I20260925 14:49:29.737800 7851 ts_tablet_manager.cc:1939] T e89d628905a74300a9df95b88c81fbcf P 6b61378a78c244f6ad71c8477c0c2acb: Deleting consensus metadata
I20260925 14:49:29.737995 7850 ts_tablet_manager.cc:1939] T e89d628905a74300a9df95b88c81fbcf P 43c36349d6824cedb50b403f3e96638f: Deleting consensus metadata
I20260925 14:49:29.738089 7078 catalog_manager.cc:4841] TS 6b61378a78c244f6ad71c8477c0c2acb (127.0.98.3:41305): tablet e89d628905a74300a9df95b88c81fbcf (table test-table [id=122a21e1906945daa98396fc95361b2e]) successfully deleted
I20260925 14:49:29.738260 7076 catalog_manager.cc:4841] TS 43c36349d6824cedb50b403f3e96638f (127.0.98.2:35853): tablet e89d628905a74300a9df95b88c81fbcf (table test-table [id=122a21e1906945daa98396fc95361b2e]) successfully deleted
I20260925 14:49:29.743330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 7473
W20260925 14:49:29.743304 7823 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:29.743018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41367 (local address 127.0.0.1:54510)
0925 14:49:29.743079 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:29.743091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:29.743143 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:29.743210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41367: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:49:29.744206 7078 catalog_manager.cc:4584] TS fdfad53226de440e8e44f810d606b84d (127.0.98.1:41367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e89d628905a74300a9df95b88c81fbcf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41367: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260925 14:49:29.749325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 7609
I20260925 14:49:29.754490 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 6958
I20260925 14:49:29.760394 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 7146
2026-09-25T14:49:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4796 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:29 dist-test-slave-z2s5 krb5kdc[7855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:29 dist-test-slave-z2s5 krb5kdc[7855](info): set up 2 sockets
Sep 25 14:49:29 dist-test-slave-z2s5 krb5kdc[7855](info): commencing operation
krb5kdc: starting...
W20260925 14:49:30.802971 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:30 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260925 14:49:30.917593 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39699
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:30.997038 7868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:30.997234 7868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:30.997256 7868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:30.998448 7868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:30.998515 7868 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:49:30.998546 7868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:30.998579 7868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:30.998605 7868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:30.998631 7868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:31.000056 7868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39699
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:31.000316 7868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:31.000540 7868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:31.005496 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.005504 7873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.005489 7874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:31.005879 7868 server_base.cc:1034] running on GCE node
I20260925 14:49:31.006126 7868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:31.006363 7868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:31.007561 7868 hybrid_clock.cc:648] HybridClock initialized: now 1790347771007550 us; error 34 us; skew 500 ppm
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:31.010102 7868 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260925 14:49:31.010633 7868 webserver.cc:460] Webserver started at http://127.0.98.62:38855/ using document root <none> and password file <none>
I20260925 14:49:31.010876 7868 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:31.010929 7868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:31.011072 7868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:31.012116 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "57a023ab05144a0a8d3c4c8305171e74"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "b209f111963e0af0be863f17c45693c7"
server_key_iv: "355b5a24562b86c7d4cd8bd83f292f18"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.012487 7868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "57a023ab05144a0a8d3c4c8305171e74"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "b209f111963e0af0be863f17c45693c7"
server_key_iv: "355b5a24562b86c7d4cd8bd83f292f18"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.013799 7868 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:49:31.014554 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.014745 7868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:31.014853 7868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "57a023ab05144a0a8d3c4c8305171e74"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "b209f111963e0af0be863f17c45693c7"
server_key_iv: "355b5a24562b86c7d4cd8bd83f292f18"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.014925 7868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:31.046602 7868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:31.048734 7868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:31.048907 7868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:31.053741 7868 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39699
I20260925 14:49:31.053820 7945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39699 every 8 connection(s)
I20260925 14:49:31.054141 7868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:31.054855 7946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.057106 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Bootstrap starting.
I20260925 14:49:31.057631 7946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.057875 7946 log.cc:824] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:31.058594 7946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: No bootstrap required, opened a new log
I20260925 14:49:31.060591 7946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a023ab05144a0a8d3c4c8305171e74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39699 } }
I20260925 14:49:31.060731 7946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.060752 7946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a023ab05144a0a8d3c4c8305171e74, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.060905 7946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a023ab05144a0a8d3c4c8305171e74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39699 } }
I20260925 14:49:31.060987 7946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:31.061010 7946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:31.061039 7946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.061523 7946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a023ab05144a0a8d3c4c8305171e74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39699 } }
I20260925 14:49:31.061604 7946 leader_election.cc:302] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a023ab05144a0a8d3c4c8305171e74; no voters:
I20260925 14:49:31.061767 7946 leader_election.cc:288] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:31.061841 7949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:31.061966 7949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [term 1 LEADER]: Becoming Leader. State: Replica: 57a023ab05144a0a8d3c4c8305171e74, State: Running, Role: LEADER
I20260925 14:49:31.062049 7949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a023ab05144a0a8d3c4c8305171e74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39699 } }
I20260925 14:49:31.062095 7946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:31.062436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 7868
I20260925 14:49:31.062542 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:31.062757 392 external_mini_cluster.cc:1428] Setting key 9823db3bbc1420da94ac153dee7cb9ed
I20260925 14:49:31.063288 7950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a023ab05144a0a8d3c4c8305171e74. Latest consensus state: current_term: 1 leader_uuid: "57a023ab05144a0a8d3c4c8305171e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a023ab05144a0a8d3c4c8305171e74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39699 } } }
I20260925 14:49:31.063454 7951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a023ab05144a0a8d3c4c8305171e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a023ab05144a0a8d3c4c8305171e74" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39699 } } }
I20260925 14:49:31.063377 7950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:31.063678 7951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74 [sys.catalog]: This master's current role is: LEADER
W20260925 14:49:31.063912 7966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:49:31.063968 7966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:49:31.064037 7968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:31.064183 7968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:31.065747 7968 catalog_manager.cc:1347] Generated new cluster ID: d0b6c21c09974bd1b3de81d3dd0cd309
I20260925 14:49:31.065801 7968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:31.073395 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.063255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54844 (local address 127.0.98.62:39699)
0925 14:49:31.063475 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.063484 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.063920 (+ 436us) server_negotiation.cc:407] Connection header received
0925 14:49:31.064104 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.064113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.064171 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.064334 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.065050 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.066239 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.067051 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.067476 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.067605 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.069969 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.069974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.069974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.069986 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.071746 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.072129 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.072134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.072134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.072224 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.072440 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.072441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.072442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.072696 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.072878 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.072975 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.073213 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":116,"threads_started":1}
I20260925 14:49:31.083552 7968 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:31.084287 7968 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:31.091498 7968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Generated new TSK 0
I20260925 14:49:31.091708 7968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260925 14:49:31.123476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39699
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:31.213127 7972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:31.213340 7972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:31.213366 7972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:31.214653 7972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:31.214694 7972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:31.214716 7972 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:31.214735 7972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:31.216485 7972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:31.216693 7972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:31.216877 7972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:31.222152 7980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.222149 7977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.222146 7978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:31.222380 7972 server_base.cc:1034] running on GCE node
I20260925 14:49:31.222589 7972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:31.222828 7972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:31.223990 7972 hybrid_clock.cc:648] HybridClock initialized: now 1790347771223988 us; error 42 us; skew 500 ppm
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:31.226475 7972 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:49:31.226889 7972 webserver.cc:460] Webserver started at http://127.0.98.1:38459/ using document root <none> and password file <none>
I20260925 14:49:31.227126 7972 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:31.227195 7972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:31.227327 7972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:31.228381 7972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "96fc416dc09b43299cf8795adc9c2b17"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "18960304fc7ebe1b7c64c0ed6726da0d"
server_key_iv: "90cc4029c32fcb3784de42334257f617"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.228771 7972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "96fc416dc09b43299cf8795adc9c2b17"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "18960304fc7ebe1b7c64c0ed6726da0d"
server_key_iv: "90cc4029c32fcb3784de42334257f617"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.230472 7972 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260925 14:49:31.231182 7987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.231369 7972 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:31.231462 7972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "96fc416dc09b43299cf8795adc9c2b17"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "18960304fc7ebe1b7c64c0ed6726da0d"
server_key_iv: "90cc4029c32fcb3784de42334257f617"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.231550 7972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:31.240487 7972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:31.242796 7972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:31.242960 7972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:31.243474 7972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:31.243541 7972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.243577 7972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:31.243607 7972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.250753 7972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33079
I20260925 14:49:31.250852 8102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33079 every 8 connection(s)
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:31.251161 7972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:31.253973 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.244034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36549 (local address 127.0.98.62:39699)
0925 14:49:31.244097 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.244100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.244312 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:49:31.244847 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.244848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.244866 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.244933 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.245919 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.246796 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.247796 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.248129 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.248205 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.251609 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.251611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.251612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.251622 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.252878 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.253239 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.253240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.253240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.253288 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.253467 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.253467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.253467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.253560 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.253607 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.253847 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.253911 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:49:31.254110 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.244001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.98.1:36549)
0925 14:49:31.244185 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.244228 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.244736 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.245068 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.245074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.245479 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:31.245819 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.245826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.246893 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.246894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.247691 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.247693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.248233 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.248234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.248409 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.249539 (+ 1130us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:31.249550 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:31.251522 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:31.252950 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.252956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.252957 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.253164 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.253336 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.253337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.253337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.253428 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.253615 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:31.253617 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:31.253733 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.253820 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.253931 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260925 14:49:31.254648 8103 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39699
I20260925 14:49:31.254735 8103 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:31.254918 8103 heartbeater.cc:511] Master 127.0.98.62:39699 requested a full tablet report, sending...
I20260925 14:49:31.255519 7899 ts_manager.cc:194] Registered new tserver with Master: 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079)
I20260925 14:49:31.256309 7899 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36549
I20260925 14:49:31.258715 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 7972
I20260925 14:49:31.258857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:31.259105 392 external_mini_cluster.cc:1428] Setting key 32bc292ed6549431564eeac74d0cf027
I20260925 14:49:31.261791 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.255692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44395 (local address 127.0.98.62:39699)
0925 14:49:31.255731 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.255733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.255900 (+ 167us) server_negotiation.cc:407] Connection header received
0925 14:49:31.255952 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.255953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.255963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.256018 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.256298 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.256952 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.257640 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.257817 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.257864 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.258778 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.258779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.258780 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.258787 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.260556 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.260971 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.260972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.260973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.261033 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.261244 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.261246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.261246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.261368 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.261445 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.261616 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.261693 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:49:31.261811 8109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.255653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.98.1:44395)
0925 14:49:31.255823 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.255834 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.255893 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.256021 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.256021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.256163 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:31.256210 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.256212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.257046 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.257048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.257573 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.257575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.257879 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.257879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.257923 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.258291 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:31.258295 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:31.258675 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:31.260727 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.260728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.260729 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.260893 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.261101 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.261101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.261102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.261151 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.261451 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:31.261452 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:31.261494 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.261592 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.261693 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":126,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260925 14:49:31.285790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39699
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:31.366846 8112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:31.367085 8112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:31.367126 8112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:31.368289 8112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:31.368363 8112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:31.368414 8112 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:31.368454 8112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:31.369966 8112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:31.370244 8112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:31.370505 8112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:31.375485 8117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:31.375545 8112 server_base.cc:1034] running on GCE node
W20260925 14:49:31.375716 8120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.375559 8118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:31.376014 8112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:31.376214 8112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:31.377357 8112 hybrid_clock.cc:648] HybridClock initialized: now 1790347771377341 us; error 31 us; skew 500 ppm
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:31.379963 8112 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:49:31.380429 8112 webserver.cc:460] Webserver started at http://127.0.98.2:43179/ using document root <none> and password file <none>
I20260925 14:49:31.380678 8112 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:31.380748 8112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:31.380893 8112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:31.381937 8112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "b23aad9251774df6a4b37a9fb5c416d0"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "4cdfe9485638aeb19639f0337b6c6d9a"
server_key_iv: "ceeeb02cf464cdd2db320effd057bc32"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.382319 8112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b23aad9251774df6a4b37a9fb5c416d0"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "4cdfe9485638aeb19639f0337b6c6d9a"
server_key_iv: "ceeeb02cf464cdd2db320effd057bc32"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.383643 8112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:31.384382 8127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.384574 8112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.384678 8112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "b23aad9251774df6a4b37a9fb5c416d0"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "4cdfe9485638aeb19639f0337b6c6d9a"
server_key_iv: "ceeeb02cf464cdd2db320effd057bc32"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.384752 8112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:31.404918 8112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:31.407356 8112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:31.407526 8112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:31.408048 8112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:31.408113 8112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.408160 8112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:31.408185 8112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.415086 8112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42849
I20260925 14:49:31.415192 8242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42849 every 8 connection(s)
I20260925 14:49:31.415468 8112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:31.418849 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.408605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42387 (local address 127.0.98.62:39699)
0925 14:49:31.408679 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.408682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.408897 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:49:31.409414 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.409415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.409430 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.409502 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.410351 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.411210 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.412229 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.412557 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.412631 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.416048 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.416050 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.416051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.416058 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.417462 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.417877 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.417878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.417879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.417940 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.418192 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.418193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.418194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.418310 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.418419 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.418681 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.418757 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:49:31.419050 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.408579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.98.2:42387)
0925 14:49:31.408759 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.408798 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.409287 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.409585 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.409589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.409947 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:31.410239 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.410246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.411306 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.411307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.412129 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.412131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.412676 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.412677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.412861 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.414001 (+ 1140us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:31.414022 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:31.415934 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:31.417564 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.417570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.417572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.417788 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.418006 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.418007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.418008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.418125 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.418385 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:31.418411 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:31.418549 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.418659 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.418816 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260925 14:49:31.419476 8243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39699
I20260925 14:49:31.419559 8243 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:31.419750 8243 heartbeater.cc:511] Master 127.0.98.62:39699 requested a full tablet report, sending...
I20260925 14:49:31.420140 7900 ts_manager.cc:194] Registered new tserver with Master: b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849)
I20260925 14:49:31.420665 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42387
I20260925 14:49:31.420894 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 8112
I20260925 14:49:31.421030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:31.421262 392 external_mini_cluster.cc:1428] Setting key 66f5c3627c12849bbc13da19514647b0
I20260925 14:49:31.426923 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.420499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38927 (local address 127.0.98.62:39699)
0925 14:49:31.420551 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.420552 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.420611 (+ 59us) server_negotiation.cc:407] Connection header received
0925 14:49:31.420675 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.420676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.420683 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.420778 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.421277 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.421959 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.423093 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.423306 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.423399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.424344 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.424345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.424346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.424352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.425724 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.426089 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.426090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.426090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.426149 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.426345 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.426346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.426346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.426519 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.426600 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.426756 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.426831 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:49:31.426954 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.420418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.98.2:38927)
0925 14:49:31.420556 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.420566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.420622 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.420884 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.420885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.421114 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:31.421184 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.421186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.422083 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.422084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.422993 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.422996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.423402 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.423403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.423474 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.423932 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:31.423937 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:31.424234 (+ 297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:31.425828 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.425829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.425830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.426007 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.426222 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.426223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.426224 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.426284 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.426591 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:31.426591 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:31.426636 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.426730 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.426832 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 14:49:31.445436 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39699
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:31.526716 8252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:31.526921 8252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:31.526945 8252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:31.528038 8252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:31.528072 8252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:31.528092 8252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:49:31.528110 8252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:31.529613 8252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:31.529803 8252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:31.529969 8252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:31.535065 8257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.535125 8258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:31.535065 8260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:31.535166 8252 server_base.cc:1034] running on GCE node
I20260925 14:49:31.535535 8252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:31.535785 8252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:31.536943 8252 hybrid_clock.cc:648] HybridClock initialized: now 1790347771536926 us; error 28 us; skew 500 ppm
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:31.539417 8252 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:49:31.539911 8252 webserver.cc:460] Webserver started at http://127.0.98.3:33249/ using document root <none> and password file <none>
I20260925 14:49:31.540150 8252 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:31.540220 8252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:31.540372 8252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:31.541431 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "90711464b1ae44928a161ecb86d2f2b0"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "3255e5c56f858013e64cb25491869553"
server_key_iv: "42f6e5083df2453282160853e484dda6"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.541869 8252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "90711464b1ae44928a161ecb86d2f2b0"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "3255e5c56f858013e64cb25491869553"
server_key_iv: "42f6e5083df2453282160853e484dda6"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.543326 8252 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:31.544183 8267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.544413 8252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:49:31.544528 8252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "90711464b1ae44928a161ecb86d2f2b0"
format_stamp: "Formatted at 2026-09-25 14:49:31 on dist-test-slave-z2s5"
server_key: "3255e5c56f858013e64cb25491869553"
server_key_iv: "42f6e5083df2453282160853e484dda6"
server_key_version: "encryptionkey@0"
I20260925 14:49:31.544612 8252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:31.576566 8252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:31.578722 8252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:31.578893 8252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:31.579396 8252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:31.579460 8252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.579509 8252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:31.579545 8252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:31.586777 8252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37335
I20260925 14:49:31.586874 8382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37335 every 8 connection(s)
I20260925 14:49:31.587198 8252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:31.589522 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.579937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48735 (local address 127.0.98.62:39699)
0925 14:49:31.580004 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.580007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.580204 (+ 197us) server_negotiation.cc:407] Connection header received
0925 14:49:31.580759 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.580760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.580774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.580856 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.581697 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.582566 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.583710 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.584073 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.584160 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.586421 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.586422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.586422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.586430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.588100 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.588554 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.588556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.588556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.588626 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.588888 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.588889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.588890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.589012 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.589090 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.589351 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.589431 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:49:31.589708 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.579927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.98.3:48735)
0925 14:49:31.580087 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.580129 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.580630 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.580938 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.580942 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.581296 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:31.581586 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.581593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.582662 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.582663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.583642 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.583645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.584191 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.584192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.584330 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.584855 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:31.584865 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:31.586305 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:31.588176 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.588181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.588182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.588461 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.588707 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.588708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.588708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.588817 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.589081 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:31.589087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:31.589219 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.589328 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.589484 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 14:49:31.590183 8383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39699
I20260925 14:49:31.590258 8383 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:31.590363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 8252
I20260925 14:49:31.590487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:31.590488 8383 heartbeater.cc:511] Master 127.0.98.62:39699 requested a full tablet report, sending...
I20260925 14:49:31.590678 392 external_mini_cluster.cc:1428] Setting key 187fcfef45afaa39cc66987ebbacbf79
I20260925 14:49:31.590991 7900 ts_manager.cc:194] Registered new tserver with Master: 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:31.591399 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48735
I20260925 14:49:31.592212 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:49:31.597362 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.591817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41085 (local address 127.0.98.62:39699)
0925 14:49:31.591877 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.591878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.592009 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:49:31.592106 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.592107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.592117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.592170 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.592477 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.592939 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.593636 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.593832 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.593914 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.594887 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.594888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.594888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.594895 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.596249 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.596609 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.596610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.596611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.596664 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.596867 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.596868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.596869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.596976 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.597049 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.597204 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.597269 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:49:31.597371 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.591788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.98.3:41085)
0925 14:49:31.591948 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.591959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.592035 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.592174 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.592174 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.592342 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:31.592390 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.592391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.593030 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.593031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.593544 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.593545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.593916 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.593917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.593988 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.594511 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:31.594515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:31.594783 (+ 268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:31.596358 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.596360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.596360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.596526 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.596733 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:31.596734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:31.596735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:31.596790 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:31.597042 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:31.597042 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:31.597085 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.597168 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.597266 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":86,"threads_started":1}
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:49:31.612916 7965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:31.607519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.0.98.62:39699)
0925 14:49:31.607588 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.607590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.607651 (+ 61us) server_negotiation.cc:407] Connection header received
0925 14:49:31.607734 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.607735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.607751 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.607811 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.608396 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.608881 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.609630 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.609815 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.609901 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.612822 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54856: BlockingRecv error: recv got EOF from 127.0.0.1:54856 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:49:31 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:31.620460 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.613786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.0.98.62:39699)
0925 14:49:31.613853 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.613855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.613863 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:31.613887 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.613888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.613897 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.613964 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:31.614462 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.614921 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.615606 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.615789 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.615861 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.617953 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:31.617954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:31.617955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:31.617962 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:31.619314 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.619696 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.619697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.619698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.619755 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:31.619952 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:31.619953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:31.619954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:31.620068 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:31.620176 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.620280 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.620358 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:49:31.622020 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:31.623737 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:31.630340 8406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.626109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37335 (local address 127.0.0.1:39012)
0925 14:49:31.626295 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.626340 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.626433 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.626696 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.626700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.626702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.626784 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.626785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.628123 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.628124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.629243 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.629245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.630102 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.630103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.630157 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.630168 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.630219 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.630254 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":88,"thread_start_us":75,"threads_started":1}
I20260925 14:49:31.630443 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.626284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.98.3:37335)
0925 14:49:31.626509 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.626515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.626527 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:49:31.626570 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.626572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.626589 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.626674 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.626887 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.628028 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.629329 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.630024 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.630101 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.630180 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.630243 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.630278 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":85,"threads_started":1}
I20260925 14:49:31.630955 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.626324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42849 (local address 127.0.0.1:45404)
0925 14:49:31.626763 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.626772 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.626820 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.627087 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.627088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.627090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.627131 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.627132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.628066 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.628067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.629461 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.629463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.630792 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.630792 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.630840 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.630844 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.630883 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.630911 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":107,"thread_start_us":154,"threads_started":1}
I20260925 14:49:31.631179 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd03234429d4bfb9da474a21a8bbc63 (DEFAULT_TABLE table=test-table [id=ccdf35103adf47be977998339c9ee379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:31.631408 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.626142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33079 (local address 127.0.0.1:47856)
0925 14:49:31.626529 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.626540 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.626589 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.626971 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.626972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.626973 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.627013 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.627014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.628228 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.628228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.629460 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.629463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.631191 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.631193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.631261 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.631277 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.631307 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.631344 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":11,"thread_start_us":79,"threads_started":1}
I20260925 14:49:31.631582 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd03234429d4bfb9da474a21a8bbc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.631803 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.626277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47856 (local address 127.0.98.1:33079)
0925 14:49:31.626801 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.626806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.626815 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:31.626853 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.626856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.626872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.627082 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.627138 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.628154 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.630439 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.631070 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.631573 (+ 503us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.631661 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.631687 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.631724 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":100,"threads_started":1}
I20260925 14:49:31.632203 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.626583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.0.98.2:42849)
0925 14:49:31.626745 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.626749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.626899 (+ 150us) server_negotiation.cc:407] Connection header received
0925 14:49:31.626939 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.626941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.626955 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.627026 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.627226 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.627979 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.629548 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.630264 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.630329 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.632043 (+ 1714us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.632074 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.632112 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260925 14:49:31.632405 8037 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd03234429d4bfb9da474a21a8bbc63 (DEFAULT_TABLE table=test-table [id=ccdf35103adf47be977998339c9ee379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:31.632746 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd03234429d4bfb9da474a21a8bbc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.632772 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd03234429d4bfb9da474a21a8bbc63 (DEFAULT_TABLE table=test-table [id=ccdf35103adf47be977998339c9ee379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:31.633074 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd03234429d4bfb9da474a21a8bbc63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.635092 8411 tablet_bootstrap.cc:492] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Bootstrap starting.
I20260925 14:49:31.635651 8411 tablet_bootstrap.cc:654] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.635960 8411 log.cc:824] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:31.636638 8413 tablet_bootstrap.cc:492] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Bootstrap starting.
I20260925 14:49:31.636727 8411 tablet_bootstrap.cc:492] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: No bootstrap required, opened a new log
I20260925 14:49:31.636790 8411 ts_tablet_manager.cc:1397] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:31.637219 8414 tablet_bootstrap.cc:492] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Bootstrap starting.
I20260925 14:49:31.637431 8413 tablet_bootstrap.cc:654] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.637765 8413 log.cc:824] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:31.637809 8414 tablet_bootstrap.cc:654] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.638144 8414 log.cc:824] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:31.638569 8413 tablet_bootstrap.cc:492] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: No bootstrap required, opened a new log
I20260925 14:49:31.638643 8413 ts_tablet_manager.cc:1397] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:31.638878 8414 tablet_bootstrap.cc:492] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: No bootstrap required, opened a new log
I20260925 14:49:31.638958 8414 ts_tablet_manager.cc:1397] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:31.638938 8411 raft_consensus.cc:348] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.639130 8411 raft_consensus.cc:374] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.639173 8411 raft_consensus.cc:729] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.639288 8411 consensus_queue.cc:260] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.639561 8411 ts_tablet_manager.cc:1428] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:49:31.639586 8383 heartbeater.cc:503] Master 127.0.98.62:39699 was elected leader, sending a full tablet report...
I20260925 14:49:31.640802 8413 raft_consensus.cc:348] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.640957 8413 raft_consensus.cc:374] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.640981 8413 raft_consensus.cc:729] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.641093 8413 consensus_queue.cc:260] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.641084 8414 raft_consensus.cc:348] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.641258 8414 raft_consensus.cc:374] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.641304 8414 raft_consensus.cc:729] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23aad9251774df6a4b37a9fb5c416d0, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.641353 8413 ts_tablet_manager.cc:1428] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:31.641386 8103 heartbeater.cc:503] Master 127.0.98.62:39699 was elected leader, sending a full tablet report...
I20260925 14:49:31.641420 8414 consensus_queue.cc:260] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.641680 8414 ts_tablet_manager.cc:1428] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:31.641727 8243 heartbeater.cc:503] Master 127.0.98.62:39699 was elected leader, sending a full tablet report...
W20260925 14:49:31.666047 8244 tablet.cc:2369] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:31.688403 8417 raft_consensus.cc:482] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:31.688524 8417 raft_consensus.cc:504] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.688948 8417 leader_election.cc:288] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079)
I20260925 14:49:31.692926 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.689260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40371 (local address 127.0.98.1:33079)
0925 14:49:31.689314 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.689316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.689333 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:49:31.689376 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.689377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.689389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.689450 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.689623 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.690441 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.691951 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.692508 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.692590 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.692600 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.692818 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.692856 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:49:31.692926 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.689159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37435 (local address 127.0.98.2:42849)
0925 14:49:31.689210 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.689212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.689398 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:49:31.689503 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.689504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.689523 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.689565 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.689700 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.690435 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.691951 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.692513 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.692590 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.692600 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.692818 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.692856 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:49:31.692961 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.689116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33079 (local address 127.0.98.3:40371)
0925 14:49:31.689181 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.689194 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.689242 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.689475 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.689476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.689478 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.689547 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.689549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.690521 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.690523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.691862 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.691864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.692628 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.692629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.692727 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.692732 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.692802 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.692856 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":77}
I20260925 14:49:31.692971 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.689126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42849 (local address 127.0.98.3:37435)
0925 14:49:31.689291 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.689303 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.689410 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.689569 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.689569 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.689571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.689632 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.689633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.690521 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.690523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.691861 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.691864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.692628 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.692629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.692727 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.692732 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.692802 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.692847 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":55,"thread_start_us":115,"threads_started":1}
I20260925 14:49:31.693365 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd03234429d4bfb9da474a21a8bbc63" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0" is_pre_election: true
I20260925 14:49:31.693553 8197 raft_consensus.cc:2454] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 0.
I20260925 14:49:31.693811 8269 leader_election.cc:302] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90711464b1ae44928a161ecb86d2f2b0, b23aad9251774df6a4b37a9fb5c416d0; no voters:
I20260925 14:49:31.693785 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd03234429d4bfb9da474a21a8bbc63" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc416dc09b43299cf8795adc9c2b17" is_pre_election: true
I20260925 14:49:31.693953 8057 raft_consensus.cc:2454] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 0.
I20260925 14:49:31.693944 8417 raft_consensus.cc:2781] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:31.694027 8417 raft_consensus.cc:482] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:31.694070 8417 raft_consensus.cc:3037] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.695079 8417 raft_consensus.cc:504] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.695219 8417 leader_election.cc:288] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 election: Requested vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079)
I20260925 14:49:31.695363 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd03234429d4bfb9da474a21a8bbc63" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0"
I20260925 14:49:31.695397 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd03234429d4bfb9da474a21a8bbc63" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc416dc09b43299cf8795adc9c2b17"
I20260925 14:49:31.695462 8057 raft_consensus.cc:3037] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.695461 8197 raft_consensus.cc:3037] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.696066 8057 raft_consensus.cc:2454] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 1.
I20260925 14:49:31.696202 8197 raft_consensus.cc:2454] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 1.
I20260925 14:49:31.696254 8270 leader_election.cc:302] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90711464b1ae44928a161ecb86d2f2b0, 96fc416dc09b43299cf8795adc9c2b17; no voters:
I20260925 14:49:31.696327 8417 raft_consensus.cc:2781] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:31.696516 8417 raft_consensus.cc:686] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 LEADER]: Becoming Leader. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Running, Role: LEADER
I20260925 14:49:31.696635 8417 consensus_queue.cc:237] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } }
I20260925 14:49:31.697525 7900 catalog_manager.cc:5495] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "90711464b1ae44928a161ecb86d2f2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:31.705984 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.703701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39014 (local address 127.0.98.3:37335)
0925 14:49:31.703755 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.703757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.703764 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:31.703807 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.703808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.703823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.703879 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:31.703963 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.704405 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.705175 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.705342 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.705389 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.705608 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:31.705722 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:31.705854 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.705880 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.705903 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:31.707749 8197 raft_consensus.cc:1261] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Refusing update from remote peer 90711464b1ae44928a161ecb86d2f2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:31.707749 8057 raft_consensus.cc:1261] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Refusing update from remote peer 90711464b1ae44928a161ecb86d2f2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:31.708117 8421 consensus_queue.cc:1035] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:31.708225 8417 consensus_queue.cc:1035] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:31.710546 8426 mvcc.cc:205] Tried to move back new op lower bound from 7333264472912330752 to 7333264472870166528. Current Snapshot: MvccSnapshot[applied={T|T < 7333264472912330752}]
I20260925 14:49:31.711277 8425 mvcc.cc:205] Tried to move back new op lower bound from 7333264472912330752 to 7333264472870166528. Current Snapshot: MvccSnapshot[applied={T|T < 7333264472912330752}]
I20260925 14:49:31.712728 7899 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:31.712818 7899 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:31.713552 7899 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Sending DeleteTablet for 3 replicas of tablet 1cd03234429d4bfb9da474a21a8bbc63
I20260925 14:49:31.713876 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd03234429d4bfb9da474a21a8bbc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47856
I20260925 14:49:31.713980 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd03234429d4bfb9da474a21a8bbc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260925 14:49:31.714052 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd03234429d4bfb9da474a21a8bbc63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:31 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45404
I20260925 14:49:31.714246 8436 tablet_replica.cc:330] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: stopping tablet replica
I20260925 14:49:31.714377 8436 raft_consensus.cc:2229] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:31.714430 8434 tablet_replica.cc:330] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: stopping tablet replica
I20260925 14:49:31.714336 7899 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:31.714497 7899 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:31.714514 8434 raft_consensus.cc:2229] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:31.714612 8434 raft_consensus.cc:2258] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:31.714766 8436 raft_consensus.cc:2258] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:31.714774 8435 tablet_replica.cc:330] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: stopping tablet replica
I20260925 14:49:31.714857 8435 raft_consensus.cc:2229] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:31.715056 8435 raft_consensus.cc:2258] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:31.715319 8436 ts_tablet_manager.cc:1905] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:31.715622 8037 tablet_service.cc:1467] Processing CreateTablet for tablet 0b159868b0d346e1b45a6c9027f6e3a8 (DEFAULT_TABLE table=test-table [id=4438dd40f3244fd3bfac52e613d610fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:31.715744 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b159868b0d346e1b45a6c9027f6e3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.715871 8435 ts_tablet_manager.cc:1905] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:31.717162 8413 tablet_bootstrap.cc:492] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Bootstrap starting.
I20260925 14:49:31.717571 8413 tablet_bootstrap.cc:654] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.717592 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 0b159868b0d346e1b45a6c9027f6e3a8 (DEFAULT_TABLE table=test-table [id=4438dd40f3244fd3bfac52e613d610fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:31.717656 8435 ts_tablet_manager.cc:1918] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:31.717720 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 0b159868b0d346e1b45a6c9027f6e3a8 (DEFAULT_TABLE table=test-table [id=4438dd40f3244fd3bfac52e613d610fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:31.717813 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b159868b0d346e1b45a6c9027f6e3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.718148 8436 ts_tablet_manager.cc:1918] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:31.718149 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b159868b0d346e1b45a6c9027f6e3a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:31.718325 8436 log.cc:1196] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/1cd03234429d4bfb9da474a21a8bbc63
I20260925 14:49:31.718704 8436 ts_tablet_manager.cc:1939] T 1cd03234429d4bfb9da474a21a8bbc63 P b23aad9251774df6a4b37a9fb5c416d0: Deleting consensus metadata
I20260925 14:49:31.717713 8435 log.cc:1196] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/1cd03234429d4bfb9da474a21a8bbc63
I20260925 14:49:31.719233 8435 ts_tablet_manager.cc:1939] T 1cd03234429d4bfb9da474a21a8bbc63 P 90711464b1ae44928a161ecb86d2f2b0: Deleting consensus metadata
I20260925 14:49:31.719525 8413 tablet_bootstrap.cc:492] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: No bootstrap required, opened a new log
I20260925 14:49:31.719570 8413 ts_tablet_manager.cc:1397] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:31.719583 8414 tablet_bootstrap.cc:492] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Bootstrap starting.
I20260925 14:49:31.719632 8411 tablet_bootstrap.cc:492] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Bootstrap starting.
I20260925 14:49:31.719730 8413 raft_consensus.cc:348] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.719790 8413 raft_consensus.cc:374] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.719811 8413 raft_consensus.cc:729] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.719866 8413 consensus_queue.cc:260] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.719944 8413 ts_tablet_manager.cc:1428] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.720005 8414 tablet_bootstrap.cc:654] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.720006 8411 tablet_bootstrap.cc:654] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:31.720389 7885 catalog_manager.cc:4841] TS 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335): tablet 1cd03234429d4bfb9da474a21a8bbc63 (table test-table [id=ccdf35103adf47be977998339c9ee379]) successfully deleted
I20260925 14:49:31.720484 7885 catalog_manager.cc:4841] TS b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849): tablet 1cd03234429d4bfb9da474a21a8bbc63 (table test-table [id=ccdf35103adf47be977998339c9ee379]) successfully deleted
I20260925 14:49:31.720875 8414 tablet_bootstrap.cc:492] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: No bootstrap required, opened a new log
I20260925 14:49:31.720932 8414 ts_tablet_manager.cc:1397] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:31.720983 8411 tablet_bootstrap.cc:492] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: No bootstrap required, opened a new log
I20260925 14:49:31.721036 8411 ts_tablet_manager.cc:1397] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:31.721067 8414 raft_consensus.cc:348] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.721138 8414 raft_consensus.cc:374] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.721159 8414 raft_consensus.cc:729] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23aad9251774df6a4b37a9fb5c416d0, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.721163 8411 raft_consensus.cc:348] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.721205 8411 raft_consensus.cc:374] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:31.721218 8411 raft_consensus.cc:729] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:31.721197 8414 consensus_queue.cc:260] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.721247 8411 consensus_queue.cc:260] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.721277 8414 ts_tablet_manager.cc:1428] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.721349 8411 ts_tablet_manager.cc:1428] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:31.721472 8434 ts_tablet_manager.cc:1905] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:31.722854 8434 ts_tablet_manager.cc:1918] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:31.722944 8434 log.cc:1196] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/1cd03234429d4bfb9da474a21a8bbc63
I20260925 14:49:31.723209 8434 ts_tablet_manager.cc:1939] T 1cd03234429d4bfb9da474a21a8bbc63 P 96fc416dc09b43299cf8795adc9c2b17: Deleting consensus metadata
I20260925 14:49:31.723572 7886 catalog_manager.cc:4841] TS 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079): tablet 1cd03234429d4bfb9da474a21a8bbc63 (table test-table [id=ccdf35103adf47be977998339c9ee379]) successfully deleted
W20260925 14:49:31.751947 8104 tablet.cc:2369] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:49:31.838516 8384 tablet.cc:2369] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:31.938627 8418 raft_consensus.cc:482] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:31.938723 8418 raft_consensus.cc:504] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.939107 8418 leader_election.cc:288] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:31.942113 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.939386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40829 (local address 127.0.98.2:42849)
0925 14:49:31.939456 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.939458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.939629 (+ 171us) server_negotiation.cc:407] Connection header received
0925 14:49:31.939668 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.939669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.939676 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.939739 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.939909 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.940411 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.941377 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.941739 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.941835 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.941846 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.941997 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.942033 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:49:31.942121 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.939593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54545 (local address 127.0.98.3:37335)
0925 14:49:31.939663 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:31.939665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:31.939676 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:31.939729 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:31.939730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:31.939739 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:31.939795 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:31.939942 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.940439 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.941399 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.941757 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.941859 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.941868 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:31.942019 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:49:31.942053 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:49:31.942157 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.939502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37335 (local address 127.0.98.1:54545)
0925 14:49:31.939576 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.939586 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.939670 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.939799 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.939800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.939801 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.939867 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.939869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.940526 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.940527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.941281 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.941283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.941856 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.941857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.941932 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.941935 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.942010 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.942053 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1,"thread_start_us":35,"threads_started":1}
I20260925 14:49:31.942425 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:31.939293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42849 (local address 127.0.98.1:40829)
0925 14:49:31.939452 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:31.939466 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:31.939573 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:31.939756 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:31.939757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:31.939759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:31.939839 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.939840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.940479 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.940480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.941279 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:31.941281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:31.941836 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:31.941837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:31.941919 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:31.941923 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:31.941991 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:49:31.942039 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260925 14:49:31.942447 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b159868b0d346e1b45a6c9027f6e3a8" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90711464b1ae44928a161ecb86d2f2b0" is_pre_election: true
I20260925 14:49:31.942605 8334 raft_consensus.cc:2454] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 0.
I20260925 14:49:31.942790 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b159868b0d346e1b45a6c9027f6e3a8" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0" is_pre_election: true
I20260925 14:49:31.942884 8197 raft_consensus.cc:2454] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 0.
I20260925 14:49:31.943064 7989 leader_election.cc:302] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96fc416dc09b43299cf8795adc9c2b17, b23aad9251774df6a4b37a9fb5c416d0; no voters:
I20260925 14:49:31.943188 8418 raft_consensus.cc:2781] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:31.943228 8418 raft_consensus.cc:482] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:31.943257 8418 raft_consensus.cc:3037] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.943948 8418 raft_consensus.cc:504] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.944092 8418 leader_election.cc:288] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 election: Requested vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:31.944300 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b159868b0d346e1b45a6c9027f6e3a8" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90711464b1ae44928a161ecb86d2f2b0"
I20260925 14:49:31.944293 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b159868b0d346e1b45a6c9027f6e3a8" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0"
I20260925 14:49:31.944372 8197 raft_consensus.cc:3037] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.944396 8334 raft_consensus.cc:3037] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:31.944952 8197 raft_consensus.cc:2454] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 1.
I20260925 14:49:31.945046 8334 raft_consensus.cc:2454] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 1.
I20260925 14:49:31.945107 7989 leader_election.cc:302] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96fc416dc09b43299cf8795adc9c2b17, b23aad9251774df6a4b37a9fb5c416d0; no voters:
I20260925 14:49:31.945174 8418 raft_consensus.cc:2781] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:31.945297 8418 raft_consensus.cc:686] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Running, Role: LEADER
I20260925 14:49:31.945386 8418 consensus_queue.cc:237] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:31.946064 7900 catalog_manager.cc:5495] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347771, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:32.022141 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.015186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.0.98.62:39699)
0925 14:49:32.015271 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.015273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.015425 (+ 152us) server_negotiation.cc:407] Connection header received
0925 14:49:32.015523 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.015524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.015534 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.015604 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:32.016087 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.016567 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.017316 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.017498 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.017568 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.019652 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:32.019654 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:32.019654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:32.019663 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:32.020915 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:32.021296 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:32.021298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:32.021298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:32.021355 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:32.021554 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:32.021555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:32.021555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:32.021665 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:32.021738 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.021894 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.022049 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":56}
I20260925 14:49:32.022141 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.015102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39699 (local address 127.0.0.1:54880)
0925 14:49:32.015303 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:32.015318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:32.015447 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:32.015625 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:32.015626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:32.015947 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:32.015991 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:32.015993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.016656 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.016657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:32.017224 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:32.017226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.017566 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.017567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:32.017617 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.018125 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:32.018130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:32.019533 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:32.021012 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:32.021019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:32.021021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:32.021211 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:32.021421 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:32.021422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:32.021423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:32.021480 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:32.021728 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:32.021734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:32.021779 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:49:32.021867 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:49:32.021997 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260925 14:49:32.024206 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54880:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:49:32.024421 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:32.025810 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 2c84125379b94e9684e7468f9d81f916 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=542342311ac3421c83b08ee48194f93e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:32.025800 8037 tablet_service.cc:1467] Processing CreateTablet for tablet 2c84125379b94e9684e7468f9d81f916 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=542342311ac3421c83b08ee48194f93e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:32.025806 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 2c84125379b94e9684e7468f9d81f916 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=542342311ac3421c83b08ee48194f93e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:49:32.025969 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c84125379b94e9684e7468f9d81f916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.025969 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c84125379b94e9684e7468f9d81f916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.025970 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c84125379b94e9684e7468f9d81f916. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.027321 8414 tablet_bootstrap.cc:492] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: Bootstrap starting.
I20260925 14:49:32.027467 8413 tablet_bootstrap.cc:492] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: Bootstrap starting.
I20260925 14:49:32.027470 8411 tablet_bootstrap.cc:492] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: Bootstrap starting.
I20260925 14:49:32.027760 8414 tablet_bootstrap.cc:654] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.027777 8411 tablet_bootstrap.cc:654] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.027784 8413 tablet_bootstrap.cc:654] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.028354 8411 tablet_bootstrap.cc:492] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: No bootstrap required, opened a new log
I20260925 14:49:32.028424 8411 ts_tablet_manager.cc:1397] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:32.028455 8413 tablet_bootstrap.cc:492] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: No bootstrap required, opened a new log
I20260925 14:49:32.028508 8413 ts_tablet_manager.cc:1397] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:32.028553 8411 raft_consensus.cc:348] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.028640 8411 raft_consensus.cc:374] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.028671 8411 raft_consensus.cc:729] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.028652 8413 raft_consensus.cc:348] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.028743 8413 raft_consensus.cc:374] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.028723 8411 consensus_queue.cc:260] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.028772 8413 raft_consensus.cc:729] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.028823 8411 ts_tablet_manager.cc:1428] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:32.028815 8413 consensus_queue.cc:260] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.028910 8413 ts_tablet_manager.cc:1428] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:32.028852 8430 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.028942 8430 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.029028 8418 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.029073 8414 tablet_bootstrap.cc:492] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: No bootstrap required, opened a new log
I20260925 14:49:32.029095 8418 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.029120 8414 ts_tablet_manager.cc:1397] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:32.029273 8414 raft_consensus.cc:348] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.029361 8414 raft_consensus.cc:374] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.029383 8414 raft_consensus.cc:729] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23aad9251774df6a4b37a9fb5c416d0, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.029423 8414 consensus_queue.cc:260] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.029520 8414 ts_tablet_manager.cc:1428] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:32.029588 8419 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.029717 8419 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.184641 8418 raft_consensus.cc:482] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:32.184737 8418 raft_consensus.cc:504] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.184885 8418 leader_election.cc:288] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:32.185176 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c84125379b94e9684e7468f9d81f916" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90711464b1ae44928a161ecb86d2f2b0" is_pre_election: true
I20260925 14:49:32.185173 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c84125379b94e9684e7468f9d81f916" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0" is_pre_election: true
I20260925 14:49:32.185282 8197 raft_consensus.cc:2454] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 0.
I20260925 14:49:32.185282 8334 raft_consensus.cc:2454] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 0.
I20260925 14:49:32.185447 7989 leader_election.cc:302] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90711464b1ae44928a161ecb86d2f2b0, 96fc416dc09b43299cf8795adc9c2b17; no voters:
I20260925 14:49:32.185575 8418 raft_consensus.cc:2781] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:32.185619 8418 raft_consensus.cc:482] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:32.185649 8418 raft_consensus.cc:3037] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.186303 8418 raft_consensus.cc:504] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.186450 8418 leader_election.cc:288] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 election: Requested vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:32.186668 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c84125379b94e9684e7468f9d81f916" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0"
I20260925 14:49:32.186684 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c84125379b94e9684e7468f9d81f916" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90711464b1ae44928a161ecb86d2f2b0"
I20260925 14:49:32.186753 8334 raft_consensus.cc:3037] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.186753 8197 raft_consensus.cc:3037] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.187510 8334 raft_consensus.cc:2454] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 1.
I20260925 14:49:32.187512 8197 raft_consensus.cc:2454] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 1.
I20260925 14:49:32.187700 7989 leader_election.cc:302] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96fc416dc09b43299cf8795adc9c2b17, b23aad9251774df6a4b37a9fb5c416d0; no voters:
I20260925 14:49:32.187808 8418 raft_consensus.cc:2781] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:32.187904 8418 raft_consensus.cc:686] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Running, Role: LEADER
I20260925 14:49:32.187963 8418 consensus_queue.cc:237] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.188292 8442 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: TxnStatusTablet state changed. Reason: New leader 96fc416dc09b43299cf8795adc9c2b17. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.188395 8442 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:32.188589 8454 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:49:32.188723 8454 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:49:32.188670 7899 catalog_manager.cc:5495] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:32.212147 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.209270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33079 (local address 127.0.0.1:47864)
0925 14:49:32.209370 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:32.209383 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:32.209441 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:32.209660 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:32.209661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:32.209662 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:32.209718 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:32.209720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.210492 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.210493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:32.211064 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:32.211065 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.211569 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.211570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:32.211639 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.211652 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.211955 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.211964 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 14:49:32.212022 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:49:32.212077 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 14:49:32.212153 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.209327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.0.98.1:33079)
0925 14:49:32.209455 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.209457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.209466 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:32.209545 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.209546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.209566 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.209655 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:32.209793 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.210379 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.211158 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.211505 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.211571 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.211782 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.211888 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.211973 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.212028 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.212052 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260925 14:49:32.213512 8197 raft_consensus.cc:1261] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Refusing update from remote peer 96fc416dc09b43299cf8795adc9c2b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.213577 8334 raft_consensus.cc:1261] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Refusing update from remote peer 96fc416dc09b43299cf8795adc9c2b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.213809 8418 consensus_queue.cc:1035] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.213932 8442 consensus_queue.cc:1035] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.214641 8430 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: TxnStatusTablet state changed. Reason: New leader 96fc416dc09b43299cf8795adc9c2b17. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.214738 8430 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.214892 8419 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: TxnStatusTablet state changed. Reason: New leader 96fc416dc09b43299cf8795adc9c2b17. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.214991 8419 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.215734 8442 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.215819 8442 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:32.216334 8419 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.216426 8419 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P b23aad9251774df6a4b37a9fb5c416d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.217010 8418 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.217104 8418 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 96fc416dc09b43299cf8795adc9c2b17: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:49:32.217519 8430 tablet_replica.cc:439] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } } }
I20260925 14:49:32.217609 8430 tablet_replica.cc:442] T 2c84125379b94e9684e7468f9d81f916 P 90711464b1ae44928a161ecb86d2f2b0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:49:32.220569 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.217578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.0.98.1:33079)
0925 14:49:32.217620 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.217622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.217691 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:49:32.217825 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.217826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.217836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.217864 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:32.218085 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.218755 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.219854 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.220048 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.220106 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.220347 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.220374 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.220414 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.220498 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.220516 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:32.225864 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.223373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52917 (local address 127.0.98.1:33079)
0925 14:49:32.223408 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.223410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.223585 (+ 175us) server_negotiation.cc:407] Connection header received
0925 14:49:32.223607 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.223608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.223619 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.223675 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:32.223800 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.224243 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.225275 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.225440 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.225483 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.225657 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.225680 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.225718 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.225770 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.225787 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:49:32.225956 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.223329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33079 (local address 127.0.98.1:52917)
0925 14:49:32.223489 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:32.223500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:32.223555 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:32.223676 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:32.223677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:32.223679 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:49:32.223743 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:32.223744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.224321 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.224322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:32.225191 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:32.225193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.225506 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.225507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:32.225567 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.225583 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.225718 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.225720 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:32.225768 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:49:32.225812 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260925 14:49:32.228426 8334 raft_consensus.cc:1261] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Refusing update from remote peer 96fc416dc09b43299cf8795adc9c2b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.228426 8197 raft_consensus.cc:1261] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Refusing update from remote peer 96fc416dc09b43299cf8795adc9c2b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.228658 8464 consensus_queue.cc:1035] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.228752 8418 consensus_queue.cc:1035] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.231331 8459 mvcc.cc:205] Tried to move back new op lower bound from 7333264475046240256 to 7333264473888997376. Current Snapshot: MvccSnapshot[applied={T|T < 7333264475046240256}]
I20260925 14:49:32.242466 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:32.242589 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:32.243208 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Sending DeleteTablet for 3 replicas of tablet 0b159868b0d346e1b45a6c9027f6e3a8
I20260925 14:49:32.243474 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b159868b0d346e1b45a6c9027f6e3a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45404
I20260925 14:49:32.243502 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b159868b0d346e1b45a6c9027f6e3a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260925 14:49:32.243724 8475 tablet_replica.cc:330] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: stopping tablet replica
I20260925 14:49:32.243849 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b159868b0d346e1b45a6c9027f6e3a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47856
I20260925 14:49:32.246802 8475 raft_consensus.cc:2229] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:32.246896 8475 raft_consensus.cc:2258] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.247108 8476 tablet_replica.cc:330] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: stopping tablet replica
I20260925 14:49:32.247179 8476 raft_consensus.cc:2229] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:32.247232 8476 raft_consensus.cc:2258] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.247355 8475 ts_tablet_manager.cc:1905] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.247375 8477 tablet_replica.cc:330] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: stopping tablet replica
I20260925 14:49:32.247426 8477 raft_consensus.cc:2229] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:32.247522 8477 raft_consensus.cc:2258] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.247720 8476 ts_tablet_manager.cc:1905] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.249118 8475 ts_tablet_manager.cc:1918] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:32.249185 8475 log.cc:1196] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/0b159868b0d346e1b45a6c9027f6e3a8
I20260925 14:49:32.249401 8475 ts_tablet_manager.cc:1939] T 0b159868b0d346e1b45a6c9027f6e3a8 P b23aad9251774df6a4b37a9fb5c416d0: Deleting consensus metadata
I20260925 14:49:32.249547 8476 ts_tablet_manager.cc:1918] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:32.249608 8476 log.cc:1196] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/0b159868b0d346e1b45a6c9027f6e3a8
I20260925 14:49:32.249850 8476 ts_tablet_manager.cc:1939] T 0b159868b0d346e1b45a6c9027f6e3a8 P 90711464b1ae44928a161ecb86d2f2b0: Deleting consensus metadata
I20260925 14:49:32.249959 7885 catalog_manager.cc:4841] TS b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849): tablet 0b159868b0d346e1b45a6c9027f6e3a8 (table test-table [id=4438dd40f3244fd3bfac52e613d610fc]) successfully deleted
I20260925 14:49:32.250156 7885 catalog_manager.cc:4841] TS 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335): tablet 0b159868b0d346e1b45a6c9027f6e3a8 (table test-table [id=4438dd40f3244fd3bfac52e613d610fc]) successfully deleted
I20260925 14:49:32.250357 8477 ts_tablet_manager.cc:1905] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.251605 8477 ts_tablet_manager.cc:1918] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:32.251695 8477 log.cc:1196] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/0b159868b0d346e1b45a6c9027f6e3a8
I20260925 14:49:32.251926 8477 ts_tablet_manager.cc:1939] T 0b159868b0d346e1b45a6c9027f6e3a8 P 96fc416dc09b43299cf8795adc9c2b17: Deleting consensus metadata
I20260925 14:49:32.252213 7886 catalog_manager.cc:4841] TS 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079): tablet 0b159868b0d346e1b45a6c9027f6e3a8 (table test-table [id=4438dd40f3244fd3bfac52e613d610fc]) successfully deleted
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[7855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:49:32.271050 7965 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:32.264826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.0.98.62:39699)
0925 14:49:32.264881 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.264883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.264999 (+ 116us) server_negotiation.cc:407] Connection header received
0925 14:49:32.265077 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.265078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.265087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.265139 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:32.265877 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.266353 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.267525 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.267726 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.267773 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.270949 (+ 3176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54890: BlockingRecv error: recv got EOF from 127.0.0.1:54890 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[7855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260925 14:49:32.279560 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.272098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.0.98.62:39699)
0925 14:49:32.272145 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.272147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.272181 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:49:32.272238 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.272239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.272249 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.272311 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:32.272761 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.273480 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.274152 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.274480 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.274557 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.276944 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:32.276946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:32.276946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:32.276953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:32.278345 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:32.278799 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:32.278800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:32.278800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:32.278854 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:32.279093 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:32.279093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:32.279093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:32.279194 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:32.279267 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.279422 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.279479 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:49:32.281350 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:32.281502 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:32.282951 8037 tablet_service.cc:1467] Processing CreateTablet for tablet bec5424f23164ae8a3c7177342880312 (DEFAULT_TABLE table=test-table [id=22b4c918f9ad4df28622ae95f6341d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:32.282951 8177 tablet_service.cc:1467] Processing CreateTablet for tablet bec5424f23164ae8a3c7177342880312 (DEFAULT_TABLE table=test-table [id=22b4c918f9ad4df28622ae95f6341d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:32.282954 8317 tablet_service.cc:1467] Processing CreateTablet for tablet bec5424f23164ae8a3c7177342880312 (DEFAULT_TABLE table=test-table [id=22b4c918f9ad4df28622ae95f6341d47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:32.283092 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec5424f23164ae8a3c7177342880312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.283092 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec5424f23164ae8a3c7177342880312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.283092 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bec5424f23164ae8a3c7177342880312. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.284402 8414 tablet_bootstrap.cc:492] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Bootstrap starting.
I20260925 14:49:32.284402 8413 tablet_bootstrap.cc:492] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Bootstrap starting.
I20260925 14:49:32.284842 8414 tablet_bootstrap.cc:654] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.284842 8413 tablet_bootstrap.cc:654] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.285202 8411 tablet_bootstrap.cc:492] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Bootstrap starting.
I20260925 14:49:32.285588 8411 tablet_bootstrap.cc:654] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.285619 8413 tablet_bootstrap.cc:492] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: No bootstrap required, opened a new log
I20260925 14:49:32.285665 8413 ts_tablet_manager.cc:1397] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:32.285732 8414 tablet_bootstrap.cc:492] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: No bootstrap required, opened a new log
I20260925 14:49:32.285775 8414 ts_tablet_manager.cc:1397] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:32.285818 8413 raft_consensus.cc:348] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.285890 8413 raft_consensus.cc:374] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.285912 8413 raft_consensus.cc:729] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.285923 8414 raft_consensus.cc:348] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.285957 8413 consensus_queue.cc:260] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.285996 8414 raft_consensus.cc:374] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.286023 8414 raft_consensus.cc:729] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23aad9251774df6a4b37a9fb5c416d0, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.286046 8413 ts_tablet_manager.cc:1428] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:32.286080 8414 consensus_queue.cc:260] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.286170 8414 ts_tablet_manager.cc:1428] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:32.286584 8411 tablet_bootstrap.cc:492] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: No bootstrap required, opened a new log
I20260925 14:49:32.286634 8411 ts_tablet_manager.cc:1397] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:32.286751 8411 raft_consensus.cc:348] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.286839 8411 raft_consensus.cc:374] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.286868 8411 raft_consensus.cc:729] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.286924 8411 consensus_queue.cc:260] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.287032 8411 ts_tablet_manager.cc:1428] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:32.478832 8430 raft_consensus.cc:482] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:32.478943 8430 raft_consensus.cc:504] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.479108 8430 leader_election.cc:288] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079), b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849)
I20260925 14:49:32.479324 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5424f23164ae8a3c7177342880312" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0" is_pre_election: true
I20260925 14:49:32.479411 8197 raft_consensus.cc:2454] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 0.
I20260925 14:49:32.479348 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5424f23164ae8a3c7177342880312" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc416dc09b43299cf8795adc9c2b17" is_pre_election: true
I20260925 14:49:32.479454 8057 raft_consensus.cc:2454] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 0.
I20260925 14:49:32.479594 8269 leader_election.cc:302] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90711464b1ae44928a161ecb86d2f2b0, b23aad9251774df6a4b37a9fb5c416d0; no voters:
I20260925 14:49:32.479729 8430 raft_consensus.cc:2781] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:32.479777 8430 raft_consensus.cc:482] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:32.479792 8430 raft_consensus.cc:3037] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.480487 8430 raft_consensus.cc:504] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.480620 8430 leader_election.cc:288] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 election: Requested vote from peers 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079), b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849)
I20260925 14:49:32.480741 8057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5424f23164ae8a3c7177342880312" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96fc416dc09b43299cf8795adc9c2b17"
I20260925 14:49:32.480758 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bec5424f23164ae8a3c7177342880312" candidate_uuid: "90711464b1ae44928a161ecb86d2f2b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0"
I20260925 14:49:32.480818 8197 raft_consensus.cc:3037] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.480837 8057 raft_consensus.cc:3037] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.481436 8057 raft_consensus.cc:2454] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 1.
I20260925 14:49:32.481442 8197 raft_consensus.cc:2454] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90711464b1ae44928a161ecb86d2f2b0 in term 1.
I20260925 14:49:32.481616 8270 leader_election.cc:302] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90711464b1ae44928a161ecb86d2f2b0, 96fc416dc09b43299cf8795adc9c2b17; no voters:
I20260925 14:49:32.481732 8430 raft_consensus.cc:2781] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:32.481894 8430 raft_consensus.cc:686] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 LEADER]: Becoming Leader. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Running, Role: LEADER
I20260925 14:49:32.481957 8430 consensus_queue.cc:237] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.482585 7898 catalog_manager.cc:5495] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "90711464b1ae44928a161ecb86d2f2b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } health_report { overall_health: HEALTHY } } }
I20260925 14:49:32.517064 8496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.514751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.0.98.3:37335)
0925 14:49:32.514951 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.514953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.514964 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:32.514992 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.514993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.515004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.515081 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:32.515205 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.515693 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.516404 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.516623 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.516688 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.516837 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.516871 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.516938 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.516980 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.517000 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":96,"threads_started":1}
I20260925 14:49:32.517863 8057 raft_consensus.cc:1261] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Refusing update from remote peer 90711464b1ae44928a161ecb86d2f2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.517875 8197 raft_consensus.cc:1261] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Refusing update from remote peer 90711464b1ae44928a161ecb86d2f2b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.518126 8493 consensus_queue.cc:1035] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.518224 8430 consensus_queue.cc:1035] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.520608 7900 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:32.520692 7900 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:32.521449 7900 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Sending DeleteTablet for 3 replicas of tablet bec5424f23164ae8a3c7177342880312
I20260925 14:49:32.521667 8037 tablet_service.cc:1514] Processing DeleteTablet for tablet bec5424f23164ae8a3c7177342880312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47856
I20260925 14:49:32.521667 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet bec5424f23164ae8a3c7177342880312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45404
I20260925 14:49:32.521701 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet bec5424f23164ae8a3c7177342880312 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260925 14:49:32.521761 8477 tablet_replica.cc:330] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: stopping tablet replica
I20260925 14:49:32.521790 8476 tablet_replica.cc:330] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: stopping tablet replica
I20260925 14:49:32.521817 8477 raft_consensus.cc:2229] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:32.521842 8476 raft_consensus.cc:2229] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:32.521857 8477 raft_consensus.cc:2258] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.521896 8476 raft_consensus.cc:2258] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.521917 8475 tablet_replica.cc:330] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: stopping tablet replica
I20260925 14:49:32.521967 8475 raft_consensus.cc:2229] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:32.522003 8475 raft_consensus.cc:2258] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.521929 7900 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:32.522068 7900 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:32.522361 8476 ts_tablet_manager.cc:1905] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.522816 8477 ts_tablet_manager.cc:1905] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.522943 8475 ts_tablet_manager.cc:1905] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.523365 8037 tablet_service.cc:1467] Processing CreateTablet for tablet 12f391d8ef474864bdf5a8fe31ec3cef (DEFAULT_TABLE table=test-table [id=c4611a6dd9404f57babd408cb9a5c147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:32.523464 8037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f391d8ef474864bdf5a8fe31ec3cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.524044 8476 ts_tablet_manager.cc:1918] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:32.524111 8476 log.cc:1196] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/bec5424f23164ae8a3c7177342880312
I20260925 14:49:32.524224 8477 ts_tablet_manager.cc:1918] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:32.524269 8477 log.cc:1196] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/bec5424f23164ae8a3c7177342880312
I20260925 14:49:32.524338 8476 ts_tablet_manager.cc:1939] T bec5424f23164ae8a3c7177342880312 P 90711464b1ae44928a161ecb86d2f2b0: Deleting consensus metadata
I20260925 14:49:32.524475 8477 ts_tablet_manager.cc:1939] T bec5424f23164ae8a3c7177342880312 P 96fc416dc09b43299cf8795adc9c2b17: Deleting consensus metadata
I20260925 14:49:32.524487 8475 ts_tablet_manager.cc:1918] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:32.524533 8475 log.cc:1196] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/bec5424f23164ae8a3c7177342880312
I20260925 14:49:32.524736 8475 ts_tablet_manager.cc:1939] T bec5424f23164ae8a3c7177342880312 P b23aad9251774df6a4b37a9fb5c416d0: Deleting consensus metadata
I20260925 14:49:32.524731 7885 catalog_manager.cc:4841] TS 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335): tablet bec5424f23164ae8a3c7177342880312 (table test-table [id=22b4c918f9ad4df28622ae95f6341d47]) successfully deleted
I20260925 14:49:32.524744 8413 tablet_bootstrap.cc:492] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17: Bootstrap starting.
I20260925 14:49:32.524926 8317 tablet_service.cc:1467] Processing CreateTablet for tablet 12f391d8ef474864bdf5a8fe31ec3cef (DEFAULT_TABLE table=test-table [id=c4611a6dd9404f57babd408cb9a5c147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:32.525013 8317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f391d8ef474864bdf5a8fe31ec3cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.525030 7885 catalog_manager.cc:4841] TS b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849): tablet bec5424f23164ae8a3c7177342880312 (table test-table [id=22b4c918f9ad4df28622ae95f6341d47]) successfully deleted
I20260925 14:49:32.525067 8413 tablet_bootstrap.cc:654] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.525348 7886 catalog_manager.cc:4841] TS 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079): tablet bec5424f23164ae8a3c7177342880312 (table test-table [id=22b4c918f9ad4df28622ae95f6341d47]) successfully deleted
I20260925 14:49:32.525974 8411 tablet_bootstrap.cc:492] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Bootstrap starting.
I20260925 14:49:32.526022 8177 tablet_service.cc:1467] Processing CreateTablet for tablet 12f391d8ef474864bdf5a8fe31ec3cef (DEFAULT_TABLE table=test-table [id=c4611a6dd9404f57babd408cb9a5c147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:32.526119 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12f391d8ef474864bdf5a8fe31ec3cef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:32.526278 8411 tablet_bootstrap.cc:654] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.526830 8413 tablet_bootstrap.cc:492] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17: No bootstrap required, opened a new log
I20260925 14:49:32.526882 8413 ts_tablet_manager.cc:1397] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:32.527019 8413 raft_consensus.cc:348] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.527097 8413 raft_consensus.cc:374] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.527117 8413 raft_consensus.cc:729] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.527166 8413 consensus_queue.cc:260] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.527264 8413 ts_tablet_manager.cc:1428] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:32.527339 8414 tablet_bootstrap.cc:492] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Bootstrap starting.
I20260925 14:49:32.527635 8411 tablet_bootstrap.cc:492] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: No bootstrap required, opened a new log
I20260925 14:49:32.527675 8411 ts_tablet_manager.cc:1397] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:49:32.527765 8411 raft_consensus.cc:348] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.527823 8411 raft_consensus.cc:374] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.527844 8411 raft_consensus.cc:729] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90711464b1ae44928a161ecb86d2f2b0, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.527890 8411 consensus_queue.cc:260] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.527976 8411 ts_tablet_manager.cc:1428] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:32.527971 8414 tablet_bootstrap.cc:654] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:32.528576 8414 tablet_bootstrap.cc:492] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: No bootstrap required, opened a new log
I20260925 14:49:32.528645 8414 ts_tablet_manager.cc:1397] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:32.528811 8414 raft_consensus.cc:348] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.528880 8414 raft_consensus.cc:374] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:32.528915 8414 raft_consensus.cc:729] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b23aad9251774df6a4b37a9fb5c416d0, State: Initialized, Role: FOLLOWER
I20260925 14:49:32.528959 8414 consensus_queue.cc:260] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.529057 8414 ts_tablet_manager.cc:1428] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260925 14:49:32.589193 8378 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.0.98.3:37335
I20260925 14:49:32.761684 8418 raft_consensus.cc:482] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:32.761863 8418 raft_consensus.cc:504] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.762086 8418 leader_election.cc:288] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:32.762436 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f391d8ef474864bdf5a8fe31ec3cef" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0" is_pre_election: true
I20260925 14:49:32.762565 8197 raft_consensus.cc:2454] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 0.
I20260925 14:49:32.762491 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f391d8ef474864bdf5a8fe31ec3cef" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90711464b1ae44928a161ecb86d2f2b0" is_pre_election: true
I20260925 14:49:32.762645 8334 raft_consensus.cc:2454] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 0.
I20260925 14:49:32.762748 7989 leader_election.cc:302] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96fc416dc09b43299cf8795adc9c2b17, b23aad9251774df6a4b37a9fb5c416d0; no voters:
I20260925 14:49:32.762887 8418 raft_consensus.cc:2781] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:32.762926 8418 raft_consensus.cc:482] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:32.762946 8418 raft_consensus.cc:3037] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.763651 8418 raft_consensus.cc:504] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.763788 8418 leader_election.cc:288] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 election: Requested vote from peers b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849), 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335)
I20260925 14:49:32.763933 8197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f391d8ef474864bdf5a8fe31ec3cef" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b23aad9251774df6a4b37a9fb5c416d0"
I20260925 14:49:32.763980 8334 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12f391d8ef474864bdf5a8fe31ec3cef" candidate_uuid: "96fc416dc09b43299cf8795adc9c2b17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90711464b1ae44928a161ecb86d2f2b0"
I20260925 14:49:32.764019 8197 raft_consensus.cc:3037] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.764055 8334 raft_consensus.cc:3037] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:32.764791 8334 raft_consensus.cc:2454] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 1.
I20260925 14:49:32.764784 8197 raft_consensus.cc:2454] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96fc416dc09b43299cf8795adc9c2b17 in term 1.
I20260925 14:49:32.765005 7989 leader_election.cc:302] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90711464b1ae44928a161ecb86d2f2b0, 96fc416dc09b43299cf8795adc9c2b17; no voters:
I20260925 14:49:32.765101 8418 raft_consensus.cc:2781] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:32.765214 8418 raft_consensus.cc:686] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 1 LEADER]: Becoming Leader. State: Replica: 96fc416dc09b43299cf8795adc9c2b17, State: Running, Role: LEADER
I20260925 14:49:32.765257 8418 consensus_queue.cc:237] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } }
I20260925 14:49:32.765765 7900 catalog_manager.cc:5495] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "96fc416dc09b43299cf8795adc9c2b17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96fc416dc09b43299cf8795adc9c2b17" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:32.815282 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.813009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47872 (local address 127.0.98.1:33079)
0925 14:49:32.813119 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:32.813120 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:32.813127 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:49:32.813151 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:32.813152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:32.813161 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:32.813217 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:32.813376 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.813848 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.814709 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:32.814895 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:32.814948 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:32.815057 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:32.815082 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:32.815116 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:32.815163 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 14:49:32.815178 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260925 14:49:32.817966 8197 raft_consensus.cc:1261] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Refusing update from remote peer 96fc416dc09b43299cf8795adc9c2b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.818010 8334 raft_consensus.cc:1261] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Refusing update from remote peer 96fc416dc09b43299cf8795adc9c2b17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:32.818291 8418 consensus_queue.cc:1035] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90711464b1ae44928a161ecb86d2f2b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.818481 8504 consensus_queue.cc:1035] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b23aad9251774df6a4b37a9fb5c416d0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:32.828764 7898 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:32.828923 7898 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:32.829576 7898 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57a023ab05144a0a8d3c4c8305171e74: Sending DeleteTablet for 3 replicas of tablet 12f391d8ef474864bdf5a8fe31ec3cef
I20260925 14:49:32.829838 8036 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f391d8ef474864bdf5a8fe31ec3cef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47856
I20260925 14:49:32.829886 8317 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f391d8ef474864bdf5a8fe31ec3cef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39012
I20260925 14:49:32.829886 8177 tablet_service.cc:1514] Processing DeleteTablet for tablet 12f391d8ef474864bdf5a8fe31ec3cef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:32 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45404
I20260925 14:49:32.829972 8476 tablet_replica.cc:330] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: stopping tablet replica
I20260925 14:49:32.829998 8477 tablet_replica.cc:330] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17: stopping tablet replica
I20260925 14:49:32.830025 8476 raft_consensus.cc:2229] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:32.830076 8476 raft_consensus.cc:2258] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.830070 8477 raft_consensus.cc:2229] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:32.830134 8477 raft_consensus.cc:2258] T 12f391d8ef474864bdf5a8fe31ec3cef P 96fc416dc09b43299cf8795adc9c2b17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.830152 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 7972
I20260925 14:49:32.830219 8475 tablet_replica.cc:330] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: stopping tablet replica
I20260925 14:49:32.830277 8475 raft_consensus.cc:2229] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:32.830317 8475 raft_consensus.cc:2258] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:32.832087 8476 ts_tablet_manager.cc:1905] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.832233 8475 ts_tablet_manager.cc:1905] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:32.833681 8476 ts_tablet_manager.cc:1918] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:32.833750 8476 log.cc:1196] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/12f391d8ef474864bdf5a8fe31ec3cef
I20260925 14:49:32.833952 8476 ts_tablet_manager.cc:1939] T 12f391d8ef474864bdf5a8fe31ec3cef P 90711464b1ae44928a161ecb86d2f2b0: Deleting consensus metadata
I20260925 14:49:32.834110 8475 ts_tablet_manager.cc:1918] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:49:32.834167 8475 log.cc:1196] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/12f391d8ef474864bdf5a8fe31ec3cef
I20260925 14:49:32.834261 7885 catalog_manager.cc:4841] TS 90711464b1ae44928a161ecb86d2f2b0 (127.0.98.3:37335): tablet 12f391d8ef474864bdf5a8fe31ec3cef (table test-table [id=c4611a6dd9404f57babd408cb9a5c147]) successfully deleted
I20260925 14:49:32.834421 8475 ts_tablet_manager.cc:1939] T 12f391d8ef474864bdf5a8fe31ec3cef P b23aad9251774df6a4b37a9fb5c416d0: Deleting consensus metadata
I20260925 14:49:32.834844 7885 catalog_manager.cc:4841] TS b23aad9251774df6a4b37a9fb5c416d0 (127.0.98.2:42849): tablet 12f391d8ef474864bdf5a8fe31ec3cef (table test-table [id=c4611a6dd9404f57babd408cb9a5c147]) successfully deleted
W20260925 14:49:32.837598 7886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33079 (error 108)
I20260925 14:49:32.837884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 8112
I20260925 14:49:32.839828 8512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:32.837929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33079 (local address 127.0.0.1:47880)
0925 14:49:32.839706 (+ 1777us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:32.839765 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1756,"thread_start_us":63,"threads_started":1}
W20260925 14:49:32.840010 7886 catalog_manager.cc:4584] TS 96fc416dc09b43299cf8795adc9c2b17 (127.0.98.1:33079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12f391d8ef474864bdf5a8fe31ec3cef: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33079: connect: Connection refused (error 111)
I20260925 14:49:32.842780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 8252
I20260925 14:49:32.848474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 7868
2026-09-25T14:49:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3087 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[8515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[8515](info): set up 2 sockets
Sep 25 14:49:32 dist-test-slave-z2s5 krb5kdc[8515](info): commencing operation
krb5kdc: starting...
W20260925 14:49:33.891258 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:33 dist-test-slave-z2s5 krb5kdc[8515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:34.010267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:34.091411 8528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:34.091626 8528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:34.091650 8528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:34.092829 8528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:34.092872 8528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:34.092886 8528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:34.092944 8528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:34.092969 8528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:34.094718 8528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:34.094970 8528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:34.095198 8528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:34.100000 8536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:34.100201 8533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:34.100113 8528 server_base.cc:1034] running on GCE node
W20260925 14:49:34.100000 8534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:34.100556 8528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:34.100860 8528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:34.102103 8528 hybrid_clock.cc:648] HybridClock initialized: now 1790347774102079 us; error 39 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:49:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1258 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:34 dist-test-slave-z2s5 krb5kdc[8541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:34 dist-test-slave-z2s5 krb5kdc[8541](info): set up 2 sockets
Sep 25 14:49:34 dist-test-slave-z2s5 krb5kdc[8541](info): commencing operation
krb5kdc: starting...
W20260925 14:49:35.150962 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:35.273954 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:35.353296 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:35.353521 8554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:35.353543 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:35.354692 8554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:35.354733 8554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:35.354748 8554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:35.354760 8554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:35.354772 8554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:35.356232 8554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:35.356436 8554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:35.356621 8554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:35.361418 8562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.361425 8560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.361438 8559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:35.361721 8554 server_base.cc:1034] running on GCE node
I20260925 14:49:35.361986 8554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:35.362280 8554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:35.363445 8554 hybrid_clock.cc:648] HybridClock initialized: now 1790347775363443 us; error 39 us; skew 500 ppm
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:35.369800 8554 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:35.370285 8554 webserver.cc:460] Webserver started at http://127.0.98.62:41317/ using document root <none> and password file <none>
I20260925 14:49:35.370548 8554 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:35.370599 8554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:35.370697 8554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:35.371770 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "90c69da3da0342d5b218228802845dd0"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "ad95b877b58692a6b487c24b6026b902"
server_key_iv: "c085943cd1dc9dce2ded7a7836a21401"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.372092 8554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "90c69da3da0342d5b218228802845dd0"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "ad95b877b58692a6b487c24b6026b902"
server_key_iv: "c085943cd1dc9dce2ded7a7836a21401"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.373418 8554 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:35.374125 8569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.374313 8554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:35.374442 8554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "90c69da3da0342d5b218228802845dd0"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "ad95b877b58692a6b487c24b6026b902"
server_key_iv: "c085943cd1dc9dce2ded7a7836a21401"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.374523 8554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:35.391837 8554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:35.393877 8554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:35.394001 8554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:35.397895 8554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40775
I20260925 14:49:35.397965 8621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40775 every 8 connection(s)
I20260925 14:49:35.398512 8554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:35.399222 8622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:35.401373 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Bootstrap starting.
I20260925 14:49:35.401892 8622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:35.402230 8622 log.cc:824] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:35.402928 8622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: No bootstrap required, opened a new log
I20260925 14:49:35.404426 8622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:35.404559 8622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:35.404582 8622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c69da3da0342d5b218228802845dd0, State: Initialized, Role: FOLLOWER
I20260925 14:49:35.404716 8622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:35.404798 8622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:35.404821 8622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:35.404850 8622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:35.405691 8622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:35.405776 8622 leader_election.cc:302] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90c69da3da0342d5b218228802845dd0; no voters:
I20260925 14:49:35.405902 8622 leader_election.cc:288] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:35.405990 8625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:35.406154 8625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 LEADER]: Becoming Leader. State: Replica: 90c69da3da0342d5b218228802845dd0, State: Running, Role: LEADER
I20260925 14:49:35.406162 8622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:35.406273 8625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:35.406670 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90c69da3da0342d5b218228802845dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } } }
I20260925 14:49:35.406675 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90c69da3da0342d5b218228802845dd0. Latest consensus state: current_term: 1 leader_uuid: "90c69da3da0342d5b218228802845dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } } }
I20260925 14:49:35.406803 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:35.406813 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:35.407063 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:35.407575 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:35.408727 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 8554
I20260925 14:49:35.408834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:35.409039 392 external_mini_cluster.cc:1428] Setting key 87bf925d9facb88c9eade8614a0c9328
I20260925 14:49:35.409061 8634 catalog_manager.cc:1347] Generated new cluster ID: 2640cba32c3c46a1bcf66105cadc0f2c
I20260925 14:49:35.409103 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:35.416066 8634 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:35.416620 8634 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:35.419565 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Generated new TSK 0
I20260925 14:49:35.419755 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:35.420639 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.409499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.0.98.62:40775)
0925 14:49:35.409669 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:35.409679 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:35.409731 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:49:35.409839 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:35.409846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:35.409897 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:35.410020 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:35.410855 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.412289 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.413369 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.413986 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.414079 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.416920 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:35.416924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:35.416924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:35.416937 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:35.418913 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.419321 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.419324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.419325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.419433 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.419739 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.419741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.419741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.420015 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:35.420189 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:35.420254 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:49:35.420398 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:35.450613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40775
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:35.532065 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:35.532269 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:35.532294 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:35.533445 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:35.533491 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:35.533519 8648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:35.535130 8648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:35.535378 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:35.535634 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:35.540658 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.540658 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.540711 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:35.541260 8648 server_base.cc:1034] running on GCE node
I20260925 14:49:35.541436 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:35.541643 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:35.542778 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790347775542760 us; error 36 us; skew 500 ppm
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:35.545217 8648 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:35.545672 8648 webserver.cc:460] Webserver started at http://127.0.98.1:36803/ using document root <none> and password file <none>
I20260925 14:49:35.545903 8648 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:35.545949 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:35.546114 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:35.547262 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "625dfd222b594f8d9b3eae76e4848faa"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "1d7616e1547448325bd6db340eaf8bb0"
server_key_iv: "1817b1ec54e65e657d09a05cb62449dc"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.547631 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "625dfd222b594f8d9b3eae76e4848faa"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "1d7616e1547448325bd6db340eaf8bb0"
server_key_iv: "1817b1ec54e65e657d09a05cb62449dc"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.549005 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:35.549860 8663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.550060 8648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:35.550169 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "625dfd222b594f8d9b3eae76e4848faa"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "1d7616e1547448325bd6db340eaf8bb0"
server_key_iv: "1817b1ec54e65e657d09a05cb62449dc"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.550277 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:35.562798 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:35.565384 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:35.565569 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:35.565824 8648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:35.566156 8648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:35.566216 8648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.566260 8648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:35.566295 8648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.572813 8648 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36345
I20260925 14:49:35.572850 8776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36345 every 8 connection(s)
I20260925 14:49:35.573222 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:35.575510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 8648
I20260925 14:49:35.575661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:35.575904 392 external_mini_cluster.cc:1428] Setting key 375c3ccb7e5e621871fcf11e2485a19a
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:49:35.583990 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.574042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49089 (local address 127.0.98.62:40775)
0925 14:49:35.574114 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:35.574116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:35.574442 (+ 326us) server_negotiation.cc:407] Connection header received
0925 14:49:35.575339 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:35.575340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:35.575356 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:35.575521 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:35.576380 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.576940 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.578275 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.578541 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.578607 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.581087 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:35.581089 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:35.581089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:35.581096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:35.582614 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.583028 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.583029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.583030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.583087 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.583334 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.583335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.583336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.583459 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:35.583544 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:35.583815 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 14:49:35.583898 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:49:35.584209 8779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.573970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:49089)
0925 14:49:35.574153 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:35.574194 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:35.575186 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:35.575619 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:35.575624 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:35.575982 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:35.576282 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:35.576287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.577028 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.577030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:35.578178 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:35.578181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.578634 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.578635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:35.578782 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.579342 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:35.579352 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:35.580986 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:35.582694 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:35.582701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:35.582704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:35.582929 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:35.583154 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:35.583155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:35.583155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:35.583265 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:35.583534 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:35.583540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:35.583683 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:49:35.583794 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:49:35.583956 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":103,"threads_started":1}
I20260925 14:49:35.584748 8777 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40775
I20260925 14:49:35.584856 8777 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:35.585080 8777 heartbeater.cc:511] Master 127.0.98.62:40775 requested a full tablet report, sending...
I20260925 14:49:35.585577 8586 ts_manager.cc:194] Registered new tserver with Master: 625dfd222b594f8d9b3eae76e4848faa (127.0.98.1:36345)
I20260925 14:49:35.586184 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:35.601801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40775
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:35.681638 8782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:35.681890 8782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:35.681927 8782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:35.683123 8782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:35.683194 8782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:35.683240 8782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:35.684775 8782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:35.685036 8782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:35.685261 8782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:35.690421 8787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.690502 8788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.690416 8790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:35.690768 8782 server_base.cc:1034] running on GCE node
I20260925 14:49:35.691007 8782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:35.691246 8782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:35.692396 8782 hybrid_clock.cc:648] HybridClock initialized: now 1790347775692385 us; error 27 us; skew 500 ppm
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:35.694880 8782 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:35.695369 8782 webserver.cc:460] Webserver started at http://127.0.98.2:36573/ using document root <none> and password file <none>
I20260925 14:49:35.695637 8782 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:35.695709 8782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:35.695866 8782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:35.696971 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "e57808ff1d78417a8f463a89b8bdf8c1"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "9b4d8b0f6a6b89470d3fb68af29ac6cb"
server_key_iv: "3c71469347db3d4b25eb1833f7e69a6e"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.697345 8782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e57808ff1d78417a8f463a89b8bdf8c1"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "9b4d8b0f6a6b89470d3fb68af29ac6cb"
server_key_iv: "3c71469347db3d4b25eb1833f7e69a6e"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.698725 8782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:35.699440 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.699597 8782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:35.699690 8782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "e57808ff1d78417a8f463a89b8bdf8c1"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "9b4d8b0f6a6b89470d3fb68af29ac6cb"
server_key_iv: "3c71469347db3d4b25eb1833f7e69a6e"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.699771 8782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:35.711970 8782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:35.713781 8782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:35.713953 8782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:35.714190 8782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:35.714546 8782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:35.714600 8782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.714641 8782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:35.714668 8782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.720355 8782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40065
I20260925 14:49:35.720399 8910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40065 every 8 connection(s)
I20260925 14:49:35.720726 8782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:35.726374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 8782
I20260925 14:49:35.726562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:35.726755 392 external_mini_cluster.cc:1428] Setting key b167a1254041a36d27159ca0d8b0ece1
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:35.730741 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.721463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39135 (local address 127.0.98.62:40775)
0925 14:49:35.721511 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:35.721514 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:35.721705 (+ 191us) server_negotiation.cc:407] Connection header received
0925 14:49:35.722716 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:35.722717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:35.722728 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:35.722791 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:35.723420 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.723939 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.724913 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.725111 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.725184 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.727558 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:35.727559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:35.727560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:35.727567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:35.729174 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.729669 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.729670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.729670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.729722 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.730011 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.730012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.730012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.730115 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:35.730169 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:35.730511 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 14:49:35.730602 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:35.730783 8913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.721430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:39135)
0925 14:49:35.721592 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:35.721634 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:35.722617 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:35.722869 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:35.722872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:35.723130 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:35.723332 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:35.723335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.724016 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.724017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:35.724821 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:35.724824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.725198 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.725198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:35.725294 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.725824 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:35.725832 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:35.727433 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:35.729251 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:35.729259 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:35.729261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:35.729591 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:35.729790 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:35.729791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:35.729792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:35.729941 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:35.730180 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:35.730185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:35.730331 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:49:35.730484 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 14:49:35.730621 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260925 14:49:35.731350 8911 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40775
I20260925 14:49:35.731460 8911 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:35.731665 8911 heartbeater.cc:511] Master 127.0.98.62:40775 requested a full tablet report, sending...
I20260925 14:49:35.732029 8586 ts_manager.cc:194] Registered new tserver with Master: e57808ff1d78417a8f463a89b8bdf8c1 (127.0.98.2:40065)
I20260925 14:49:35.732396 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39135
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:35.753943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40775
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:35.835255 8916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:35.835469 8916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:35.835494 8916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:35.836697 8916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:35.836745 8916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:35.836772 8916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:35.838332 8916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:35.838613 8916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:35.838868 8916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:35.843915 8924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:35.844024 8921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:35.844070 8916 server_base.cc:1034] running on GCE node
W20260925 14:49:35.844089 8922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:35.844434 8916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:35.844689 8916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:35.845847 8916 hybrid_clock.cc:648] HybridClock initialized: now 1790347775845844 us; error 38 us; skew 500 ppm
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:35.848511 8916 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:35.848973 8916 webserver.cc:460] Webserver started at http://127.0.98.3:44901/ using document root <none> and password file <none>
I20260925 14:49:35.849229 8916 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:35.849278 8916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:35.849411 8916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:35.850549 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d60f586227c4925b975880184ac3a95"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "e33a3919d30fac894eb09af0a6ca9795"
server_key_iv: "cf0692dd5dd7075d8feb6cced5bec4f0"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.850905 8916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d60f586227c4925b975880184ac3a95"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "e33a3919d30fac894eb09af0a6ca9795"
server_key_iv: "cf0692dd5dd7075d8feb6cced5bec4f0"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.852888 8916 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 14:49:35.853694 8931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.853865 8916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:35.853964 8916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "1d60f586227c4925b975880184ac3a95"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "e33a3919d30fac894eb09af0a6ca9795"
server_key_iv: "cf0692dd5dd7075d8feb6cced5bec4f0"
server_key_version: "encryptionkey@0"
I20260925 14:49:35.854046 8916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:35.890457 8916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:35.893006 8916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:35.893218 8916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:35.893472 8916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:35.893812 8916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:35.893868 8916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.893910 8916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:35.893939 8916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:35.900084 8916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35593
I20260925 14:49:35.900126 9044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35593 every 8 connection(s)
I20260925 14:49:35.900480 8916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:49:35 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:35.909420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 8916
I20260925 14:49:35.909582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:35.909821 392 external_mini_cluster.cc:1428] Setting key c9101333f92586a3649ab0da8ce0bdbf
I20260925 14:49:35.910683 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.901321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33025 (local address 127.0.98.62:40775)
0925 14:49:35.901395 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:35.901404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:35.901590 (+ 186us) server_negotiation.cc:407] Connection header received
0925 14:49:35.902595 (+ 1005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:35.902597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:35.902608 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:35.902665 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:35.903261 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.903864 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.904911 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.905108 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.905176 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.907778 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:35.907779 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:35.907779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:35.907786 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:35.909036 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.909474 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.909475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.909476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.909542 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:35.909891 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:35.909892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:35.909892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:35.910043 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:35.910110 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:35.910498 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 14:49:35.910584 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:49:35.910902 9047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:35.901314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:33025)
0925 14:49:35.901467 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:35.901504 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:35.902501 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:35.902724 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:35.902728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:35.902986 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:35.903192 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:35.903195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.903955 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.903955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:35.904785 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:35.904787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:35.905194 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:35.905195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:35.905299 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:35.905894 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:35.905902 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:35.907689 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:35.909133 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:35.909140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:35.909142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:35.909366 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:35.909626 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:35.909627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:35.909627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:35.909790 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:35.910119 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:35.910125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:35.910302 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:49:35.910474 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 14:49:35.910677 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 14:49:35.911363 9045 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40775
I20260925 14:49:35.911477 9045 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:35.911685 9045 heartbeater.cc:511] Master 127.0.98.62:40775 requested a full tablet report, sending...
I20260925 14:49:35.912024 8586 ts_manager.cc:194] Registered new tserver with Master: 1d60f586227c4925b975880184ac3a95 (127.0.98.3:35593)
I20260925 14:49:35.912361 8586 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33025
I20260925 14:49:35.921945 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260925 14:49:35.996817 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 8648
I20260925 14:49:36.002022 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 8782
I20260925 14:49:36.006582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 8916
I20260925 14:49:36.010967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 8554
I20260925 14:49:36.016036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40775
--webserver_interface=127.0.98.62
--webserver_port=41317
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:36.096479 9054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:36.096745 9054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:36.096782 9054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:36.097936 9054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:36.098016 9054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:36.098042 9054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:36.098075 9054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:36.098099 9054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:36.099759 9054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41317
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:36.100054 9054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:36.100291 9054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:36.105185 9062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:36.105201 9060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.105259 9054 server_base.cc:1034] running on GCE node
W20260925 14:49:36.105185 9059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.105667 9054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:36.105971 9054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:36.107138 9054 hybrid_clock.cc:648] HybridClock initialized: now 1790347776107137 us; error 40 us; skew 500 ppm
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:36.109624 9054 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:36.110107 9054 webserver.cc:460] Webserver started at http://127.0.98.62:41317/ using document root <none> and password file <none>
I20260925 14:49:36.110340 9054 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:36.110445 9054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:36.111896 9054 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:36.112599 9069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.112793 9054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:36.112906 9054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "90c69da3da0342d5b218228802845dd0"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "ad95b877b58692a6b487c24b6026b902"
server_key_iv: "c085943cd1dc9dce2ded7a7836a21401"
server_key_version: "encryptionkey@0"
I20260925 14:49:36.113262 9054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:36.124043 9054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:36.126042 9054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:36.126209 9054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:36.130241 9054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40775
I20260925 14:49:36.130273 9121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40775 every 8 connection(s)
I20260925 14:49:36.130833 9054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:36.132041 9122 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:49:36.132694 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Bootstrap starting.
I20260925 14:49:36.140636 9122 log.cc:824] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:36.141034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9054
I20260925 14:49:36.141399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36345
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40775
--webserver_port=36803
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:49:36.143002 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:49:36.143286 9122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Bootstrap complete.
I20260925 14:49:36.145612 9122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:36.145854 9122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c69da3da0342d5b218228802845dd0, State: Initialized, Role: FOLLOWER
I20260925 14:49:36.146067 9122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:36.146164 9122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:36.146217 9122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:36.146265 9122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:49:36.147349 9122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:36.147476 9122 leader_election.cc:302] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90c69da3da0342d5b218228802845dd0; no voters:
I20260925 14:49:36.147696 9122 leader_election.cc:288] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:49:36.147737 9126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:49:36.147848 9126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [term 2 LEADER]: Becoming Leader. State: Replica: 90c69da3da0342d5b218228802845dd0, State: Running, Role: LEADER
I20260925 14:49:36.147934 9126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } }
I20260925 14:49:36.147971 9122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:36.149165 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90c69da3da0342d5b218228802845dd0. Latest consensus state: current_term: 2 leader_uuid: "90c69da3da0342d5b218228802845dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } } }
I20260925 14:49:36.149353 9127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "90c69da3da0342d5b218228802845dd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c69da3da0342d5b218228802845dd0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40775 } } }
I20260925 14:49:36.149536 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:36.149557 9127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:36.149948 9140 catalog_manager.cc:1259] Loaded cluster ID: 2640cba32c3c46a1bcf66105cadc0f2c
I20260925 14:49:36.149992 9140 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: loading cluster ID for follower catalog manager: success
I20260925 14:49:36.151262 9140 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: acquiring CA information for follower catalog manager: success
I20260925 14:49:36.151603 9140 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:49:36.151700 9141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:36.151855 9141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:36.151938 9141 catalog_manager.cc:1259] Loaded cluster ID: 2640cba32c3c46a1bcf66105cadc0f2c
I20260925 14:49:36.151970 9141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:36.152154 9141 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:36.152235 9141 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90c69da3da0342d5b218228802845dd0: Loaded TSK: 0
I20260925 14:49:36.152357 9141 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:49:36.224989 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:36.225247 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:36.225291 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:36.226668 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:36.226735 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:36.226786 9125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:36.228422 9125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36803
--tserver_master_addrs=127.0.98.62:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:36.228708 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:36.228946 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:36.235451 9148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:36.235543 9149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.235637 9125 server_base.cc:1034] running on GCE node
W20260925 14:49:36.235451 9151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.235955 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:36.236260 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:36.237429 9125 hybrid_clock.cc:648] HybridClock initialized: now 1790347776237415 us; error 35 us; skew 500 ppm
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:36.240219 9125 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260925 14:49:36.240749 9125 webserver.cc:460] Webserver started at http://127.0.98.1:36803/ using document root <none> and password file <none>
I20260925 14:49:36.240985 9125 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:36.241066 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:36.242522 9125 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:36.243153 9158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.243309 9125 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:36.243423 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "625dfd222b594f8d9b3eae76e4848faa"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "1d7616e1547448325bd6db340eaf8bb0"
server_key_iv: "1817b1ec54e65e657d09a05cb62449dc"
server_key_version: "encryptionkey@0"
I20260925 14:49:36.243784 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:36.257519 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:36.259932 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:36.260076 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:36.260363 9125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:36.260735 9125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:36.260771 9125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.260794 9125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:36.260838 9125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.268296 9125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36345
I20260925 14:49:36.268339 9271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36345 every 8 connection(s)
I20260925 14:49:36.268685 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.276798 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9125
I20260925 14:49:36.277132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40065
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40775
--webserver_port=36573
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.278524 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.269426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:46135)
0925 14:49:36.269636 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.269686 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.270337 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.270826 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.270831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.271632 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.271924 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.271930 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.273026 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.273027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.274444 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.274447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.275028 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.275029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.275188 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.275740 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.275751 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.278272 (+ 2521us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
W20260925 14:49:36.278877 9272 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40775 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40775: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260925 14:49:36.279208 9275 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.269551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46135 (local address 127.0.98.62:40775)
0925 14:49:36.269723 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.269732 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.269770 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:49:36.270575 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.270581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.270632 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.270746 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.272041 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.272921 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.274533 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.274936 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.275018 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.278462 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46135: BlockingRecv error: recv got EOF from 127.0.98.1:46135 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.283584 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.279157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:41707)
0925 14:49:36.279234 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.279246 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.279297 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.279532 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.279533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.279676 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.279723 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.279725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.280361 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.280362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.280867 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.280869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.281229 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.281230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.281294 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.281807 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.281811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.283505 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 14:49:36.283669 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.279218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41707 (local address 127.0.98.62:40775)
0925 14:49:36.279408 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.279410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.279416 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:49:36.279436 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.279437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.279447 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.279781 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.279809 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.280271 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.280964 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.281158 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.281232 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.283589 (+ 2357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41707: BlockingRecv error: recv got EOF from 127.0.98.1:41707 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":98,"threads_started":1}
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.288048 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.284036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:35567)
0925 14:49:36.284076 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.284086 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.284120 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.284242 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.284243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.284374 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.284399 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.284400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.285006 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.285007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.285491 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.285493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.285872 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.285873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.285923 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.286350 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.286355 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.287968 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:49:36.288089 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.284081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35567 (local address 127.0.98.62:40775)
0925 14:49:36.284142 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.284144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.284155 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:36.284187 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.284188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.284196 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.284241 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.284474 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.284929 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.285585 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.285791 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.285864 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.288033 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35567: BlockingRecv error: recv got EOF from 127.0.98.1:35567 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 14:49:36.288403 9272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:49:36.366832 9276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:36.367046 9276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:36.367069 9276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:36.368201 9276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:36.368237 9276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:36.368263 9276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:36.369654 9276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40065
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36573
--tserver_master_addrs=127.0.98.62:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:36.369845 9276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:36.370028 9276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:36.375303 9276 server_base.cc:1034] running on GCE node
W20260925 14:49:36.375259 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:36.375259 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:36.375253 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.375769 9276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:36.376036 9276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:36.377206 9276 hybrid_clock.cc:648] HybridClock initialized: now 1790347776377209 us; error 46 us; skew 500 ppm
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:36.379786 9276 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260925 14:49:36.380246 9276 webserver.cc:460] Webserver started at http://127.0.98.2:36573/ using document root <none> and password file <none>
I20260925 14:49:36.380474 9276 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:36.380556 9276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:36.381920 9276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:36.382539 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.382707 9276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:36.382805 9276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "e57808ff1d78417a8f463a89b8bdf8c1"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "9b4d8b0f6a6b89470d3fb68af29ac6cb"
server_key_iv: "3c71469347db3d4b25eb1833f7e69a6e"
server_key_version: "encryptionkey@0"
I20260925 14:49:36.383186 9276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:36.403644 9276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:36.406167 9276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:36.406342 9276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:36.406623 9276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:36.406939 9276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:36.406994 9276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.407037 9276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:36.407069 9276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.412814 9276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40065
I20260925 14:49:36.412889 9405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40065 every 8 connection(s)
I20260925 14:49:36.413260 9276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:36.413826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9276
I20260925 14:49:36.414147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35593
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40775
--webserver_port=44901
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:49:36.421729 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.414045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55219 (local address 127.0.98.62:40775)
0925 14:49:36.414123 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.414126 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.414748 (+ 622us) server_negotiation.cc:407] Connection header received
0925 14:49:36.415222 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.415223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.415240 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.415324 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.416464 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.416965 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.418089 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.418292 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.418379 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.421680 (+ 3301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55219: BlockingRecv error: recv got EOF from 127.0.98.2:55219 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:49:36.421844 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.414047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:55219)
0925 14:49:36.414597 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.414641 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.415128 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.415476 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.415481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.416072 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.416299 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.416304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.417070 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.417071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.417965 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.417968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.418411 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.418412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.418549 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.419142 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.419151 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.421596 (+ 2445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":451,"threads_started":1}
W20260925 14:49:36.422091 9406 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40775 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40775: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.426782 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.422326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:46971)
0925 14:49:36.422420 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.422432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.422479 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.422626 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.422627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.422761 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.422803 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.422804 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.423443 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.423444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.423959 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.423961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.424353 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.424354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.424427 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.424892 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.424897 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.426696 (+ 1799us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 14:49:36.426833 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.422412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46971 (local address 127.0.98.62:40775)
0925 14:49:36.422458 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.422460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.422469 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:36.422543 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.422544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.422555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.422624 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.422882 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.423360 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.424067 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.424268 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.424353 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.426768 (+ 2415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46971: BlockingRecv error: recv got EOF from 127.0.98.2:46971 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.431772 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.427205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:39001)
0925 14:49:36.427256 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.427267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.427329 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.427475 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.427476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.427601 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.427628 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.427629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.428274 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.428275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.428771 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.428773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.429228 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.429228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.429282 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.429788 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.429793 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.431669 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:49:36.431965 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.427275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39001 (local address 127.0.98.62:40775)
0925 14:49:36.427331 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.427333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.427342 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:36.427390 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.427391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.427400 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.427473 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.427711 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.428193 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.428933 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.429139 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.429226 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.431778 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39001: BlockingRecv error: recv got EOF from 127.0.98.2:39001 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 14:49:36.431986 9406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:49:36.498054 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:36.498265 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:36.498288 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:36.499684 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:36.499729 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:36.499755 9409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:36.501179 9409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44901
--tserver_master_addrs=127.0.98.62:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:36.501407 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:36.501682 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:36.506652 9415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.506781 9409 server_base.cc:1034] running on GCE node
W20260925 14:49:36.506681 9417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:36.506887 9414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:36.507079 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:36.507336 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:36.508496 9409 hybrid_clock.cc:648] HybridClock initialized: now 1790347776508469 us; error 40 us; skew 500 ppm
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347776, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:36.511157 9409 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260925 14:49:36.511623 9409 webserver.cc:460] Webserver started at http://127.0.98.3:44901/ using document root <none> and password file <none>
I20260925 14:49:36.511808 9409 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:36.511861 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:36.513218 9409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:36.513753 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.513944 9409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.514014 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "1d60f586227c4925b975880184ac3a95"
format_stamp: "Formatted at 2026-09-25 14:49:35 on dist-test-slave-z2s5"
server_key: "e33a3919d30fac894eb09af0a6ca9795"
server_key_iv: "cf0692dd5dd7075d8feb6cced5bec4f0"
server_key_version: "encryptionkey@0"
I20260925 14:49:36.514340 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:36.533653 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:36.536028 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:36.536145 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:36.536342 9409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:36.536588 9409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:36.536624 9409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.536648 9409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:36.536665 9409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:36.542850 9409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35593
I20260925 14:49:36.542896 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35593 every 8 connection(s)
I20260925 14:49:36.543239 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:36.549463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9409
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:49:36.551735 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.543979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50905 (local address 127.0.98.62:40775)
0925 14:49:36.544047 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.544049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.544305 (+ 256us) server_negotiation.cc:407] Connection header received
0925 14:49:36.544845 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.544847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.544859 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.544911 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.545893 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.546536 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.547716 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.547994 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.548070 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.551674 (+ 3604us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50905: BlockingRecv error: recv got EOF from 127.0.98.3:50905 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:49:36.551748 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.543969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:50905)
0925 14:49:36.544178 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.544216 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.544733 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.544990 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.545000 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.545575 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.545774 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.545778 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.546629 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.546630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.547619 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.547621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.548075 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.548076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.548205 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.548709 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.548717 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.551579 (+ 2862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
W20260925 14:49:36.552124 9538 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40775 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:40775: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.557513 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.552365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:49343)
0925 14:49:36.552432 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.552443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.552498 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.552621 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.552621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.552741 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.552785 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.552786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.553523 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.553524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.554360 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.554363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.554998 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.555000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.555075 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.555454 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.555458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.557433 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 14:49:36.557655 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.552418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49343 (local address 127.0.98.62:40775)
0925 14:49:36.552458 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.552459 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.552487 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:49:36.552543 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.552543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.552552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.552616 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.552850 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.553424 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.554517 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.554835 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.554919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.557557 (+ 2638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49343: BlockingRecv error: recv got EOF from 127.0.98.3:49343 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:36.558981 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.550115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.0.98.62:40775)
0925 14:49:36.550159 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.550161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.550227 (+ 66us) server_negotiation.cc:407] Connection header received
0925 14:49:36.550331 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.550332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.550348 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.550483 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.551386 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.552013 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.553064 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.553270 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.553328 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.555020 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:36.555026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:36.555026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:36.555041 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:36.556837 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:36.557243 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:36.557248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:36.557248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:36.557373 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:36.558084 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:36.558086 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:36.558086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:36.558424 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:36.558552 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:36.558822 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 14:49:36.558918 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:36 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:36.562462 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:36.557841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:57049)
0925 14:49:36.557877 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:36.557887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:36.557923 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:36.558075 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:36.558076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:36.558185 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:36.558212 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.558213 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.559097 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.559098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.559883 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:36.559886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.560250 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.560251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:36.560289 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.560795 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:36.560800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:36.562351 (+ 1551us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17}
W20260925 14:49:36.562510 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:36.557880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57049 (local address 127.0.98.62:40775)
0925 14:49:36.557954 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:36.557956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:36.557965 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:36.557987 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:36.557988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:36.558000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:36.558070 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:36.558280 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.558991 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.559967 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:36.560181 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:36.560237 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:36.562471 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57049: BlockingRecv error: recv got EOF from 127.0.98.3:57049 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 14:49:36.562752 9538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 14:49:37 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:37 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:37.293471 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:37.288815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:50189)
0925 14:49:37.288959 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:37.288970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:37.289033 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:37.289198 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:37.289198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:37.289354 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:37.289404 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:37.289405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.290044 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.290045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:37.290676 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:37.290679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.291033 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.291034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:37.291088 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:37.291666 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:37.291671 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:37.293405 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260925 14:49:37.293550 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:37.288877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50189 (local address 127.0.98.62:40775)
0925 14:49:37.289005 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:37.289007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:37.289017 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:37.289095 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:37.289096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:37.289106 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:37.289178 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:37.289466 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.289967 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.290752 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.290934 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.291020 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:37.293496 (+ 2476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50189: BlockingRecv error: recv got EOF from 127.0.98.1:50189 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
Sep 25 14:49:37 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:37 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:37.437093 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:37.432461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:42189)
0925 14:49:37.432607 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:37.432617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:37.432688 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:37.432908 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:37.432909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:37.433101 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:37.433149 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:37.433151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.433799 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.433800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:37.434344 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:37.434345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.434769 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.434770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:37.434815 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:37.435338 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:37.435342 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:37.437020 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
W20260925 14:49:37.437155 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:37.432523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42189 (local address 127.0.98.62:40775)
0925 14:49:37.432580 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:37.432582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:37.432695 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:49:37.432784 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:37.432785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:37.432798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:37.432875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:37.433214 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.433729 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.434472 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.434690 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.434757 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:37.437110 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42189: BlockingRecv error: recv got EOF from 127.0.98.2:42189 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 14:49:37 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:37 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:37.567667 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:37.563170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:38821)
0925 14:49:37.563343 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:37.563353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:37.563403 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:37.563504 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:37.563505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:37.563688 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:37.563735 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:37.563737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.564403 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.564404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:37.564961 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:37.564963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.565292 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.565293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:37.565331 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:37.565775 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:37.565779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:37.567598 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
W20260925 14:49:37.567790 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:37.563234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38821 (local address 127.0.98.62:40775)
0925 14:49:37.563309 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:37.563311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:37.563412 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:49:37.563437 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:37.563438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:37.563449 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:37.563498 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:37.563824 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.564330 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.565038 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:37.565221 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:37.565271 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:37.567690 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38821: BlockingRecv error: recv got EOF from 127.0.98.3:38821 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 14:49:38 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:38 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:38.298679 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:38.294013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:43141)
0925 14:49:38.294182 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:38.294193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:38.294247 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:38.294372 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:38.294372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:38.294667 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:38.294717 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:38.294720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.295382 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.295382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:38.295914 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:38.295916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.296249 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.296250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:38.296289 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:38.296763 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:38.296768 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:38.298568 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":106,"threads_started":1}
W20260925 14:49:38.298729 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:38.294108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43141 (local address 127.0.98.62:40775)
0925 14:49:38.294232 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:38.294234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:38.294252 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:49:38.294293 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:38.294294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:38.294306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:38.294361 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:38.294790 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.295306 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.295986 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.296184 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.296236 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:38.298659 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43141: BlockingRecv error: recv got EOF from 127.0.98.1:43141 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":56,"threads_started":1}
Sep 25 14:49:38 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:38 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:38.442200 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:38.437622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:42607)
0925 14:49:38.437763 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:38.437774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:38.437830 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:38.437955 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:38.437955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:38.438123 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:38.438172 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:38.438174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.438817 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.438818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:38.439360 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:38.439362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.439692 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.439693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:38.439733 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:38.440189 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:38.440193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:38.442096 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260925 14:49:38.442284 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:38.437761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42607 (local address 127.0.98.62:40775)
0925 14:49:38.437816 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:38.437818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:38.437827 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:38.437879 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:38.437880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:38.437890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:38.437940 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:38.438242 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.438749 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.439444 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.439627 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.439686 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:38.442207 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42607: BlockingRecv error: recv got EOF from 127.0.98.2:42607 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:49:38 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:38 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:38.573201 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:38.568198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:50429)
0925 14:49:38.568349 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:38.568367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:38.568441 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:38.568579 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:38.568580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:38.568761 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:38.568804 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:38.568805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.569488 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.569490 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:38.570031 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:38.570032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.570435 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.570436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:38.570498 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:38.571007 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:38.571011 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:38.573141 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
W20260925 14:49:38.573325 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:38.568389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50429 (local address 127.0.98.62:40775)
0925 14:49:38.568470 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:38.568472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:38.568482 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:38.568509 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:38.568510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:38.568523 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:38.568575 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:38.568890 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.569396 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.570127 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:38.570326 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:38.570431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:38.573224 (+ 2793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50429: BlockingRecv error: recv got EOF from 127.0.98.3:50429 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 14:49:39 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:39 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:39.304023 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:39.299199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:56173)
0925 14:49:39.299367 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:39.299386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:39.299439 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:39.299650 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:39.299652 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:39.299832 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:39.299879 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:39.299881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.300584 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.300584 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:39.301104 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:39.301105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.301481 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.301482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:39.301520 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:39.302040 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:39.302045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:39.303959 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
W20260925 14:49:39.304159 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:39.299238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56173 (local address 127.0.98.62:40775)
0925 14:49:39.299360 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:39.299364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:39.299432 (+ 68us) server_negotiation.cc:407] Connection header received
0925 14:49:39.299534 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:39.299535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:39.299558 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:39.299630 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:39.299997 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.300510 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.301199 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.301410 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.301474 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:39.304063 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56173: BlockingRecv error: recv got EOF from 127.0.98.1:56173 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":65,"threads_started":1}
Sep 25 14:49:39 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:39 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:39.447427 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:39.442775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:43227)
0925 14:49:39.442942 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:39.442954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:39.443029 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:39.443162 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:39.443163 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:39.443326 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:39.443376 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:39.443378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.444021 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.444022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:39.444571 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:39.444572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.444941 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.444942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:39.444980 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:39.445403 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:39.445408 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:39.447335 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
W20260925 14:49:39.447503 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:39.442845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43227 (local address 127.0.98.62:40775)
0925 14:49:39.442924 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:39.442926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:39.443039 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:49:39.443075 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:39.443076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:39.443087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:39.443140 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:39.443451 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.443931 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.444645 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.444878 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.444929 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:39.447443 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43227: BlockingRecv error: recv got EOF from 127.0.98.2:43227 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 14:49:39 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:39 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:39.578795 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:39.573764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:41245)
0925 14:49:39.573926 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:39.573959 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:39.574038 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:39.574200 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:39.574201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:39.574440 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:39.574506 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:39.574508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.575222 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.575223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:39.575759 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:39.575760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.576119 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.576120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:39.576158 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:39.576653 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:39.576657 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:39.578686 (+ 2029us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
W20260925 14:49:39.578835 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:39.573785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41245 (local address 127.0.98.62:40775)
0925 14:49:39.573832 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:39.573834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:39.574054 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:49:39.574094 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:39.574095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:39.574112 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:39.574180 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:39.574636 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.575135 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.575859 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:39.576050 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:39.576109 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:39.578770 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41245: BlockingRecv error: recv got EOF from 127.0.98.3:41245 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 25 14:49:40 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:40 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:40.309535 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:40.304615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:59751)
0925 14:49:40.304773 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:40.304785 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:40.304855 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:40.305004 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:40.305005 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:40.305175 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:40.305253 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:40.305255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.306184 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.306185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:40.306793 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:40.306795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.307121 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.307121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:40.307159 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:40.307775 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:40.307779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:40.309476 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":105,"threads_started":1}
W20260925 14:49:40.309676 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:40.304711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59751 (local address 127.0.98.62:40775)
0925 14:49:40.304859 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:40.304861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:40.304902 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:49:40.304924 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:40.304925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:40.304935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:40.304985 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:40.305368 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.306115 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.306865 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.307055 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.307110 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:40.309585 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59751: BlockingRecv error: recv got EOF from 127.0.98.1:59751 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":96,"threads_started":1}
Sep 25 14:49:40 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:40 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:40.458263 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:40.447886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:48235)
0925 14:49:40.448006 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:40.448018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:40.448083 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:40.448236 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:40.448237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:40.448392 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:40.448440 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:40.448442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.449324 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.449325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:40.449891 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:40.449893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.450230 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.450230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:40.450278 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:40.456256 (+ 5978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:40.456261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:40.458177 (+ 1916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
W20260925 14:49:40.458429 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:40.447950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48235 (local address 127.0.98.62:40775)
0925 14:49:40.448032 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:40.448034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:40.448068 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:49:40.448147 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:40.448148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:40.448160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:40.448230 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:40.448543 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.449237 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.449957 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.450164 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.450219 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:40.458340 (+ 8121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48235: BlockingRecv error: recv got EOF from 127.0.98.2:48235 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:49:40 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:40 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:40.583812 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:40.579300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:36425)
0925 14:49:40.579431 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:40.579448 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:40.579503 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:40.579612 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:40.579613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:40.579783 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:40.579828 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:40.579829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.580445 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.580446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:40.580968 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:40.580969 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.581284 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.581285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:40.581323 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:40.581849 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:40.581854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:40.583748 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
W20260925 14:49:40.583940 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:40.579367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36425 (local address 127.0.98.62:40775)
0925 14:49:40.579422 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:40.579424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:40.579507 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:49:40.579544 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:40.579544 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:40.579556 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:40.579601 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:40.579887 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.580375 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.581038 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:40.581214 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:40.581264 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:40.583839 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36425: BlockingRecv error: recv got EOF from 127.0.98.3:36425 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 25 14:49:41 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:41 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:41.314254 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:41.309933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:39603)
0925 14:49:41.310079 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:41.310090 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:41.310155 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:41.310293 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:41.310294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:41.310505 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:41.310553 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:41.310555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.311193 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.311193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:41.311728 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:41.311730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.312047 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.312048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:41.312085 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:41.312500 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:41.312505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:41.314192 (+ 1687us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
W20260925 14:49:41.314363 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:41.310012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39603 (local address 127.0.98.62:40775)
0925 14:49:41.310111 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:41.310113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:41.310150 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:49:41.310210 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:41.310211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:41.310223 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:41.310273 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:41.310622 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.311126 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.311799 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.311984 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.312034 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:41.314292 (+ 2258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39603: BlockingRecv error: recv got EOF from 127.0.98.1:39603 (error 108)
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":54,"threads_started":1}
Sep 25 14:49:41 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:41 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:41.463711 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:41.458883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:46481)
0925 14:49:41.459039 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:41.459051 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:41.459135 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:41.459304 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:41.459305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:41.459484 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:41.459531 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:41.459533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.460214 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.460215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:41.460765 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:41.460767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.461165 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.461165 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:41.461233 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:41.461772 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:41.461779 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:41.463615 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
W20260925 14:49:41.463762 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:41.459064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46481 (local address 127.0.98.62:40775)
0925 14:49:41.459144 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:41.459146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:41.459157 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:41.459199 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:41.459200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:41.459231 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:41.459297 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:41.459622 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.460121 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.460858 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.461057 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.461148 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:41.463697 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46481: BlockingRecv error: recv got EOF from 127.0.98.2:46481 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:49:41 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:41 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:41.589082 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:41.584325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:33707)
0925 14:49:41.584537 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:41.584550 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:41.584639 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:41.584797 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:41.584798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:41.584971 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:41.585016 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:41.585018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.585766 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.585767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:41.586373 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:41.586375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.586801 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.586802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:41.586859 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:41.587333 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:41.587339 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:41.589023 (+ 1684us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":122,"threads_started":1}
W20260925 14:49:41.589196 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:41.584438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33707 (local address 127.0.98.62:40775)
0925 14:49:41.584523 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:41.584526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:41.584619 (+ 93us) server_negotiation.cc:407] Connection header received
0925 14:49:41.584703 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:41.584704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:41.584724 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:41.584780 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:41.585103 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.585654 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.586499 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:41.586710 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:41.586801 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:41.589119 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33707: BlockingRecv error: recv got EOF from 127.0.98.3:33707 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 14:49:42 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:42 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:42.319731 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:42.314785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:33779)
0925 14:49:42.314921 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:42.314940 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:42.314995 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:42.315139 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:42.315140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:42.315308 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:42.315356 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:42.315358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.316198 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.316198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:42.316768 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:42.316771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.317290 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.317291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:42.317353 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:42.317831 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:42.317835 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:42.319618 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260925 14:49:42.319779 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:42.314874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33779 (local address 127.0.98.62:40775)
0925 14:49:42.315014 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:42.315016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:42.315025 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:42.315046 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:42.315046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:42.315061 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:42.315123 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:42.315456 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.316115 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.316866 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.317166 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.317287 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:42.319708 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33779: BlockingRecv error: recv got EOF from 127.0.98.1:33779 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
Sep 25 14:49:42 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:42 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:42.469221 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:42.464319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:45365)
0925 14:49:42.464446 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:42.464462 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:42.464513 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:42.464661 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:42.464662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:42.464822 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:42.464869 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:42.464870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.465545 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.465546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:42.466067 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:42.466068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.466555 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.466557 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:42.466609 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:42.467109 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:42.467113 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:42.469154 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260925 14:49:42.469389 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:42.464435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45365 (local address 127.0.98.62:40775)
0925 14:49:42.464501 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:42.464505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:42.464522 (+ 17us) server_negotiation.cc:407] Connection header received
0925 14:49:42.464558 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:42.464559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:42.464572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:42.464648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:42.464937 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.465475 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.466124 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.466472 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.466529 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:42.469291 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45365: BlockingRecv error: recv got EOF from 127.0.98.2:45365 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:49:42 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:42 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:42.594805 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:42.589637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:54577)
0925 14:49:42.589866 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:42.589880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:42.589984 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:42.590175 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:42.590177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:42.590367 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:42.590438 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:42.590440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.591186 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.591187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:42.591771 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:42.591773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.592164 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.592165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:42.592227 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:42.592786 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:42.592792 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:42.594689 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":99,"threads_started":1}
W20260925 14:49:42.594842 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:42.589751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54577 (local address 127.0.98.62:40775)
0925 14:49:42.589853 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:42.589856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:42.589962 (+ 106us) server_negotiation.cc:407] Connection header received
0925 14:49:42.590064 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:42.590064 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:42.590086 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:42.590154 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:42.590580 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.591089 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.591872 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:42.592072 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:42.592165 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:42.594773 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54577: BlockingRecv error: recv got EOF from 127.0.98.3:54577 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 14:49:43 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:43 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:43.325493 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:43.320260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:33013)
0925 14:49:43.320400 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:43.320409 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:43.320460 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:43.320683 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:43.320684 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:43.320839 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:43.320890 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:43.320892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.321624 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.321625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:43.322193 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:43.322195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.322658 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.322659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:43.322713 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:43.323306 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:43.323311 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:43.325429 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
W20260925 14:49:43.325598 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:43.320317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33013 (local address 127.0.98.62:40775)
0925 14:49:43.320454 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:43.320456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:43.320467 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:43.320525 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:43.320526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:43.320537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:43.320613 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:43.320994 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.321524 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.322296 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.322529 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.322597 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:43.325524 (+ 2927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33013: BlockingRecv error: recv got EOF from 127.0.98.1:33013 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
Sep 25 14:49:43 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:43 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:43.474817 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:43.469784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:46383)
0925 14:49:43.470006 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:43.470018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:43.470117 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:43.470281 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:43.470282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:43.470532 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:43.470622 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:43.470624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.471333 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.471335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:43.471875 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:43.471877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.472245 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.472246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:43.472307 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:43.472814 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:43.472819 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:43.474715 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":91,"threads_started":1}
W20260925 14:49:43.474865 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:43.469881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46383 (local address 127.0.98.62:40775)
0925 14:49:43.469979 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:43.469981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:43.470098 (+ 117us) server_negotiation.cc:407] Connection header received
0925 14:49:43.470177 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:43.470178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:43.470193 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:43.470263 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:43.470709 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.471234 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.471958 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.472162 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.472237 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:43.474801 (+ 2564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46383: BlockingRecv error: recv got EOF from 127.0.98.2:46383 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 14:49:43 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:43 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:43.600466 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:43.595390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:56169)
0925 14:49:43.595545 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:43.595562 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:43.595630 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:43.595865 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:43.595866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:43.596089 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:43.596156 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:43.596158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.596914 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.596915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:43.597460 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:43.597462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.597873 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.597873 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:43.597914 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:43.598444 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:43.598456 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:43.600405 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
W20260925 14:49:43.600636 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:43.595498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56169 (local address 127.0.98.62:40775)
0925 14:49:43.595579 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:43.595581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:43.595616 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:49:43.595721 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:43.595722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:43.595734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:43.595831 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:43.596247 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.596835 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.597600 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:43.597786 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:43.597860 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:43.600535 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56169: BlockingRecv error: recv got EOF from 127.0.98.3:56169 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 25 14:49:44 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:44 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:44.330876 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:44.326037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:33871)
0925 14:49:44.326208 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:44.326218 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:44.326272 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:44.326499 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:44.326500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:44.326665 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:44.326715 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:44.326716 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.327383 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.327385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:44.327927 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:44.327929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.328265 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.328266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:44.328308 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:44.328822 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:44.328827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:44.330813 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
W20260925 14:49:44.331048 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:44.326201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33871 (local address 127.0.98.62:40775)
0925 14:49:44.326321 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:44.326323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:44.326332 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:44.326359 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:44.326360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:44.326370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:44.326488 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:44.326803 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.327306 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.328009 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.328200 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.328252 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:44.330942 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33871: BlockingRecv error: recv got EOF from 127.0.98.1:33871 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":79,"threads_started":1}
Sep 25 14:49:44 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:44 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:44.480261 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:44.475378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:41889)
0925 14:49:44.475596 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:44.475608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:44.475721 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:44.475881 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:44.475882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:44.476065 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:44.476109 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:44.476111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.476914 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.476915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:44.477438 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:44.477440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.477807 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.477808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:44.477865 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:44.478310 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:44.478314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:44.480205 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":108,"threads_started":1}
W20260925 14:49:44.480386 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:44.475497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41889 (local address 127.0.98.62:40775)
0925 14:49:44.475593 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:44.475595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:44.475702 (+ 107us) server_negotiation.cc:407] Connection header received
0925 14:49:44.475783 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:44.475784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:44.475797 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:44.475875 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:44.476199 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.476828 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.477531 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.477724 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.477797 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:44.480289 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41889: BlockingRecv error: recv got EOF from 127.0.98.2:41889 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 14:49:44 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:44 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:44.605748 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:44.601162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:42535)
0925 14:49:44.601293 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:44.601312 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:44.601373 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:44.601498 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:44.601498 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:44.601664 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:44.601712 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:44.601713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.602423 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.602425 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:44.603013 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:44.603015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.603354 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.603355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:44.603397 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:44.603953 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:44.603958 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:44.605689 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":70,"threads_started":1}
W20260925 14:49:44.605911 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:44.601242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42535 (local address 127.0.98.62:40775)
0925 14:49:44.601330 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:44.601332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:44.601355 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:49:44.601429 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:44.601429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:44.601443 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:44.601486 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:44.601764 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.602260 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.603089 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:44.603272 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:44.603339 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:44.605825 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42535: BlockingRecv error: recv got EOF from 127.0.98.3:42535 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 14:49:45 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:45 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:45.336038 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:45.331422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:57739)
0925 14:49:45.331589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:45.331605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:45.331660 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:45.331824 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:45.331824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:45.331991 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:45.332036 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:45.332038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.332680 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.332681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:45.333197 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:45.333199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.333554 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.333555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:45.333592 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:45.334034 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:45.334039 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:45.335941 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":70,"threads_started":1}
W20260925 14:49:45.336112 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:45.331468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57739 (local address 127.0.98.62:40775)
0925 14:49:45.331629 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:45.331631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:45.331645 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:49:45.331722 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:45.331723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:45.331733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:45.331795 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:45.332116 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.332606 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.333287 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.333484 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.333544 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:45.336028 (+ 2484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57739: BlockingRecv error: recv got EOF from 127.0.98.1:57739 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
Sep 25 14:49:45 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:45 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:45.489248 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:45.480720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:41367)
0925 14:49:45.480907 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:45.480919 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:45.481010 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:45.481186 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:45.481187 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:45.481363 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:45.481408 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:45.481410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.482082 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.482083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:45.482690 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:45.482692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.483063 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.483064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:45.483122 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:45.487282 (+ 4160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:45.487287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:45.489153 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
W20260925 14:49:45.489310 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:45.480849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41367 (local address 127.0.98.62:40775)
0925 14:49:45.480921 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:45.480923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:45.480992 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:49:45.481094 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:45.481095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:45.481107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:45.481182 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:45.481488 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.481992 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.482780 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.482977 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.483061 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:45.489236 (+ 6175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41367: BlockingRecv error: recv got EOF from 127.0.98.2:41367 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 14:49:45 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:45 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:45.611238 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:45.606285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:59379)
0925 14:49:45.606447 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:45.606462 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:45.606530 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:45.606744 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:45.606745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:45.606933 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:45.606979 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:45.606981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.607692 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.607693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:45.608236 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:45.608237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.608604 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.608605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:45.608677 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:45.609182 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:45.609192 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:45.611180 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
W20260925 14:49:45.611366 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:45.606430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59379 (local address 127.0.98.62:40775)
0925 14:49:45.606485 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:45.606487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:45.606513 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:49:45.606621 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:45.606622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:45.606635 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:45.606733 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:45.607064 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.607594 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.608326 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:45.608519 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:45.608605 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:45.611267 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59379: BlockingRecv error: recv got EOF from 127.0.98.3:59379 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 14:49:46 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:46 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:46.341727 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:46.336648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:56505)
0925 14:49:46.336789 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:46.336799 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:46.336853 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:46.337017 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:46.337017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:46.337169 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:46.337226 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:46.337227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.338239 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.338240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:46.338954 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:46.338956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.339290 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.339291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:46.339346 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:46.339868 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:46.339873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:46.341668 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260925 14:49:46.341907 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:46.336717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56505 (local address 127.0.98.62:40775)
0925 14:49:46.336875 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:46.336876 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:46.336887 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:46.336911 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:46.336912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:46.336923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:46.337005 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:46.337311 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.338163 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.339029 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.339221 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.339282 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:46.341823 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56505: BlockingRecv error: recv got EOF from 127.0.98.1:56505 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
Sep 25 14:49:46 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:46 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:46.494688 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:46.489884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:46591)
0925 14:49:46.490046 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:46.490057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:46.490153 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:46.490300 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:46.490301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:46.490524 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:46.490571 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:46.490573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.491241 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.491242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:46.491844 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:46.491846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.492202 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.492203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:46.492261 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:46.492816 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:46.492822 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:46.494590 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
W20260925 14:49:46.494735 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:46.489904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46591 (local address 127.0.98.62:40775)
0925 14:49:46.489971 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:46.489973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:46.490118 (+ 145us) server_negotiation.cc:407] Connection header received
0925 14:49:46.490211 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:46.490212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:46.490225 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:46.490294 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:46.490654 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.491154 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.491939 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.492132 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.492204 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:46.494673 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46591: BlockingRecv error: recv got EOF from 127.0.98.2:46591 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 14:49:46 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:46 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:46.616019 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:46.611683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:55495)
0925 14:49:46.611857 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:46.611877 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:46.611928 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:46.612061 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:46.612061 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:46.612216 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:46.612275 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:46.612277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.612881 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.612881 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:46.613392 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:46.613394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.613697 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.613698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:46.613734 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:46.614223 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:46.614228 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:46.615949 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":56,"threads_started":1}
W20260925 14:49:46.616175 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:46.611881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55495 (local address 127.0.98.62:40775)
0925 14:49:46.611941 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:46.611943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:46.611952 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:49:46.611980 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:46.611980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:46.611991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:46.612040 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:46.612339 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.612816 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.613462 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:46.613636 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:46.613685 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:46.616106 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55495: BlockingRecv error: recv got EOF from 127.0.98.3:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:49:47 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:47 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:47.347043 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:47.342191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:36011)
0925 14:49:47.342352 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:47.342362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:47.342473 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:47.342774 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:47.342774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:47.342935 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:47.342983 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:47.342984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.343618 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.343619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:47.344131 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:47.344133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.344507 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.344507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:47.344548 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:47.345018 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:47.345022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:47.346946 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
W20260925 14:49:47.347101 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:47.342267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36011 (local address 127.0.98.62:40775)
0925 14:49:47.342617 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:47.342619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:47.342629 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:47.342650 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:47.342651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:47.342661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:47.342762 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:47.343053 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.343551 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.344217 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.344426 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.344490 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:47.347029 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36011: BlockingRecv error: recv got EOF from 127.0.98.1:36011 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":197,"threads_started":1}
Sep 25 14:49:47 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:47 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:47.500051 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:47.495151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:53979)
0925 14:49:47.495311 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:47.495322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:47.495413 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:47.495547 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:47.495548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:47.495727 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:47.495796 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:47.495798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.496471 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.496472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:47.497005 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:47.497006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.497436 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.497437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:47.497501 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:47.498045 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:47.498049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:47.499944 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
W20260925 14:49:47.500095 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:47.495222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53979 (local address 127.0.98.62:40775)
0925 14:49:47.495305 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:47.495307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:47.495393 (+ 86us) server_negotiation.cc:407] Connection header received
0925 14:49:47.495479 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:47.495479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:47.495491 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:47.495544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:47.495881 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.496386 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.497113 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.497323 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.497435 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:47.500029 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53979: BlockingRecv error: recv got EOF from 127.0.98.2:53979 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 14:49:47 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:47 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:47.621121 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:47.616597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:55597)
0925 14:49:47.616783 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:47.616794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:47.616896 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:47.617064 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:47.617065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:47.617284 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:47.617327 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:47.617329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.617985 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.617986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:47.618530 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:47.618532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.618866 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.618867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:47.618906 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:47.619322 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:47.619327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:47.621064 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":105,"threads_started":1}
W20260925 14:49:47.621392 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:47.616690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55597 (local address 127.0.98.62:40775)
0925 14:49:47.616736 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:47.616737 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:47.616872 (+ 135us) server_negotiation.cc:407] Connection header received
0925 14:49:47.616951 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:47.616952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:47.616964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:47.617028 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:47.617388 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.617922 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.618597 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:47.618784 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:47.618846 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:47.621320 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55597: BlockingRecv error: recv got EOF from 127.0.98.3:55597 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 25 14:49:48 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:48 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:48.352174 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:48.347592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:46819)
0925 14:49:48.347744 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:48.347754 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:48.347810 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:48.347988 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:48.347989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:48.348143 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:48.348201 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:48.348203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.348912 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.348913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:48.349497 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:48.349499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.349832 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.349833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:48.349872 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:48.350381 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:48.350410 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:48.352117 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
W20260925 14:49:48.352332 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:48.347715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46819 (local address 127.0.98.62:40775)
0925 14:49:48.347834 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:48.347836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:48.347844 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:48.347863 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:48.347864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:48.347874 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:48.347982 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:48.348267 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.348846 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.349568 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.349766 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.349821 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:48.352243 (+ 2422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46819: BlockingRecv error: recv got EOF from 127.0.98.1:46819 (error 108)
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":70,"threads_started":1}
Sep 25 14:49:48 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:48 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:48.505434 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:48.500546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:34577)
0925 14:49:48.500698 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:48.500713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:48.500790 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:48.500907 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:48.500908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:48.501088 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:48.501138 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:48.501139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.501782 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.501783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:48.502327 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:48.502329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.502740 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.502741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:48.502795 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:48.503385 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:48.503390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:48.505371 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260925 14:49:48.505584 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:48.500620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34577 (local address 127.0.98.62:40775)
0925 14:49:48.500665 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:48.500668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:48.500788 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:49:48.500838 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:48.500839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:48.500849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:48.500901 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:48.501197 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.501681 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.502448 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.502643 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.502696 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:48.505498 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34577: BlockingRecv error: recv got EOF from 127.0.98.2:34577 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 14:49:48 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:48 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:48.626336 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:48.621599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:37429)
0925 14:49:48.621801 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:48.621812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:48.621891 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:48.622048 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:48.622049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:48.622228 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:48.622271 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:48.622273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.622969 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.622971 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:48.623495 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:48.623496 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.623869 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.623870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:48.623932 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:48.624355 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:48.624359 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:48.626278 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
W20260925 14:49:48.626518 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:48.621686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37429 (local address 127.0.98.62:40775)
0925 14:49:48.621765 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:48.621767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:48.621871 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:49:48.621974 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:48.621975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:48.621989 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:48.622043 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:48.622352 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.622884 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.623579 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:48.623784 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:48.623868 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:48.626452 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37429: BlockingRecv error: recv got EOF from 127.0.98.3:37429 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 14:49:49 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:49 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:49.357052 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:49.352639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:32981)
0925 14:49:49.352795 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:49.352806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:49.352869 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:49.353006 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:49.353006 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:49.353160 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:49.353207 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:49.353209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.353890 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.353891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:49.354493 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:49.354496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.354833 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.354834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:49.354875 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:49.355322 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:49.355327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:49.356987 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
W20260925 14:49:49.357151 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:49.352729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32981 (local address 127.0.98.62:40775)
0925 14:49:49.352852 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:49.352854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:49.352866 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:49:49.352925 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:49.352925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:49.352936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:49.352986 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:49.353297 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.353789 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.354581 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.354764 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.354817 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:49.357077 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32981: BlockingRecv error: recv got EOF from 127.0.98.1:32981 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":88,"threads_started":1}
Sep 25 14:49:49 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:49 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:49.510752 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:49.505958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:44141)
0925 14:49:49.506103 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:49.506113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:49.506161 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:49.506338 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:49.506338 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:49.506550 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:49.506597 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:49.506599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.507270 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.507271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:49.507799 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:49.507800 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.508215 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.508216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:49.508277 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:49.508769 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:49.508773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:49.510646 (+ 1873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
W20260925 14:49:49.510795 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:49.506151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44141 (local address 127.0.98.62:40775)
0925 14:49:49.506221 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:49.506223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:49.506231 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:49.506256 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:49.506257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:49.506268 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:49.506324 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:49.506679 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.507184 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.507931 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.508130 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.508213 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:49.510732 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44141: BlockingRecv error: recv got EOF from 127.0.98.2:44141 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 14:49:49 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:49 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:49.631996 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:49.626898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:34869)
0925 14:49:49.627065 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:49.627077 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:49.627131 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:49.627297 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:49.627298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:49.627460 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:49.627525 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:49.627527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.628245 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.628245 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:49.628767 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:49.628769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.629126 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.629126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:49.629167 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:49.629671 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:49.629677 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:49.631928 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
W20260925 14:49:49.632108 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:49.627027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34869 (local address 127.0.98.62:40775)
0925 14:49:49.627135 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:49.627137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:49.627153 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:49:49.627205 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:49.627206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:49.627221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:49.627274 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:49.627626 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.628163 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.628876 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:49.629059 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:49.629114 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:49.632016 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34869: BlockingRecv error: recv got EOF from 127.0.98.3:34869 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 14:49:50 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:50 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:50.362495 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:50.357606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:43183)
0925 14:49:50.357724 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:50.357739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:50.357788 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:50.358083 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:50.358083 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:50.358246 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:50.358296 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:50.358297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.359033 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.359033 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:50.359549 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:50.359551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.359925 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.359926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:50.359963 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:50.360427 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:50.360432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:50.362371 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260925 14:49:50.362581 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:50.357681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43183 (local address 127.0.98.62:40775)
0925 14:49:50.357839 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:50.357841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:50.357851 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:50.357965 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:50.357966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:50.357984 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:50.358068 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:50.358373 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.358958 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.359649 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.359853 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.359916 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:50.362498 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43183: BlockingRecv error: recv got EOF from 127.0.98.1:43183 (error 108)
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
Sep 25 14:49:50 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:50 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:50.520473 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:50.511265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:49085)
0925 14:49:50.511400 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:50.511410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:50.511469 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:50.511627 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:50.511628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:50.511780 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:50.511824 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:50.511826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.512533 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.512534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:50.513049 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:50.513050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.513468 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.513469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:50.513534 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:50.518606 (+ 5072us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:50.518610 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:50.520376 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
W20260925 14:49:50.520524 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:50.511355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49085 (local address 127.0.98.62:40775)
0925 14:49:50.511443 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:50.511446 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:50.511461 (+ 15us) server_negotiation.cc:407] Connection header received
0925 14:49:50.511534 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:50.511534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:50.511546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:50.511613 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:50.511921 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.512462 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.513117 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.513343 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.513459 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:50.520456 (+ 6997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49085: BlockingRecv error: recv got EOF from 127.0.98.2:49085 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:49:50 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:50 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:50.637471 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:50.632504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.3:54465)
0925 14:49:50.632695 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:50.632709 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:50.632785 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:50.632958 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:50.632958 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:50.633136 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:50.633180 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:50.633181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.634196 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.634198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:50.634756 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:50.634758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.635140 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.635140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:50.635196 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:50.635653 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:50.635658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:50.637414 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":104,"threads_started":1}
W20260925 14:49:50.637601 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:50.632571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54465 (local address 127.0.98.62:40775)
0925 14:49:50.632641 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:50.632643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:50.632767 (+ 124us) server_negotiation.cc:407] Connection header received
0925 14:49:50.632859 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:50.632860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:50.632880 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:50.632950 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:50.633279 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.634109 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.634849 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:50.635053 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:50.635139 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:50.637498 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54465: BlockingRecv error: recv got EOF from 127.0.98.3:54465 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 14:49:51 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:51 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:51.367913 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.363070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.1:53093)
0925 14:49:51.363231 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:51.363243 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:51.363307 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:51.363452 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:51.363453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:51.363609 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:51.363658 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.363659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.364335 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.364337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.364919 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.364921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.365354 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.365356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.365428 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.366067 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:51.366071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:51.367854 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
W20260925 14:49:51.368070 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:51.363140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53093 (local address 127.0.98.62:40775)
0925 14:49:51.363281 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:51.363283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:51.363293 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:51.363358 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:51.363358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:51.363370 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:51.363430 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:51.363722 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.364224 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.364997 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.365257 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.365322 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.367984 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53093: BlockingRecv error: recv got EOF from 127.0.98.1:53093 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
Sep 25 14:49:51 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:49:51 dist-test-slave-z2s5 krb5kdc[8541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:49:51.525625 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.521002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40775 (local address 127.0.98.2:50777)
0925 14:49:51.521173 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:51.521184 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:51.521272 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:51.521414 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:51.521415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:51.521588 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:51.521633 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.521634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.522300 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.522301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.522927 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.522929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.523309 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.523310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.523377 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.523842 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:51.523846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:51.525558 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:40775: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
W20260925 14:49:51.525736 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:51.521077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50777 (local address 127.0.98.62:40775)
0925 14:49:51.521156 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:51.521158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:51.521253 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:49:51.521338 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:51.521339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:51.521350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:51.521408 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:51.521725 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.522220 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.523028 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.523225 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.523308 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.525658 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50777: BlockingRecv error: recv got EOF from 127.0.98.2:50777 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:49:51.549975 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9125
I20260925 14:49:51.554498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9276
I20260925 14:49:51.558683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9409
I20260925 14:49:51.562640 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9054
2026-09-25T14:49:51Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17454 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:49:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:51Z Disabled control of system clock
I20260925 14:49:51.589702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42815
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:49:51.669279 9604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:51.669485 9604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:51.669505 9604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:51.670640 9604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:51.670680 9604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:51.670693 9604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:51.670706 9604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:51.670718 9604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:51.672000 9604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:51.672199 9604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:51.672426 9604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:51.677510 9604 server_base.cc:1034] running on GCE node
W20260925 14:49:51.677471 9609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:51.677479 9610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:51.677484 9612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:51.678102 9604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:51.678471 9604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:51.679646 9604 hybrid_clock.cc:648] HybridClock initialized: now 1790347791679615 us; error 44 us; skew 500 ppm
I20260925 14:49:51.680941 9604 webserver.cc:460] Webserver started at http://127.0.98.62:33581/ using document root <none> and password file <none>
I20260925 14:49:51.681200 9604 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:51.681272 9604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:51.681438 9604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:51.682585 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "c6598be485e94047be8d8a0279eb4595"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "32491858daec5c61905baf6acbcb2006"
server_key_iv: "2d0ecce80850d4de83ddde15e40477e5"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.682945 9604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "c6598be485e94047be8d8a0279eb4595"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "32491858daec5c61905baf6acbcb2006"
server_key_iv: "2d0ecce80850d4de83ddde15e40477e5"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.684224 9604 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:51.685071 9618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.685336 9604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:51.685458 9604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "c6598be485e94047be8d8a0279eb4595"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "32491858daec5c61905baf6acbcb2006"
server_key_iv: "2d0ecce80850d4de83ddde15e40477e5"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.685551 9604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:51.698097 9604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:51.698529 9604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:51.698711 9604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:51.703078 9604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42815
I20260925 14:49:51.703112 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42815 every 8 connection(s)
I20260925 14:49:51.703487 9604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:51.703975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9604
I20260925 14:49:51.704061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:51.704311 392 external_mini_cluster.cc:1428] Setting key 18633272f0c6764bba718540e1e10a2c
I20260925 14:49:51.704355 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:51.708112 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595: Bootstrap starting.
I20260925 14:49:51.708788 9671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:51.709138 9671 log.cc:824] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:51.710023 9671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595: No bootstrap required, opened a new log
I20260925 14:49:51.710649 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.704983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33682 (local address 127.0.98.62:42815)
0925 14:49:51.705186 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:51.705194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:51.705224 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:49:51.705951 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:51.705959 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:51.706025 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:51.706157 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:51.706499 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.707394 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.708570 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.709013 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.709108 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.709922 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:51.709925 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:51.709925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:51.710123 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:51.710164 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:51.710175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:51.710226 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:51.710331 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 14:49:51.710441 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260925 14:49:51.712178 9671 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6598be485e94047be8d8a0279eb4595" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260925 14:49:51.712308 9671 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:51.712330 9671 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6598be485e94047be8d8a0279eb4595, State: Initialized, Role: FOLLOWER
I20260925 14:49:51.712474 9671 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6598be485e94047be8d8a0279eb4595" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260925 14:49:51.712584 9671 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:51.712617 9671 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:51.712673 9671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:51.713256 9671 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6598be485e94047be8d8a0279eb4595" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260925 14:49:51.713366 9671 leader_election.cc:302] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6598be485e94047be8d8a0279eb4595; no voters:
I20260925 14:49:51.713524 9671 leader_election.cc:288] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:51.713596 9676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:51.713763 9676 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [term 1 LEADER]: Becoming Leader. State: Replica: c6598be485e94047be8d8a0279eb4595, State: Running, Role: LEADER
I20260925 14:49:51.713809 9671 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:51.713850 9676 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6598be485e94047be8d8a0279eb4595" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } }
I20260925 14:49:51.714222 9677 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6598be485e94047be8d8a0279eb4595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6598be485e94047be8d8a0279eb4595" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } } }
I20260925 14:49:51.714255 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6598be485e94047be8d8a0279eb4595. Latest consensus state: current_term: 1 leader_uuid: "c6598be485e94047be8d8a0279eb4595" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6598be485e94047be8d8a0279eb4595" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42815 } } }
I20260925 14:49:51.714414 9677 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:51.714438 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:51.715116 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:51.715507 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:51.716969 9685 catalog_manager.cc:1347] Generated new cluster ID: f3f7f3e6afc841288d17430e34cb6f9d
I20260925 14:49:51.717033 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:51.729210 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:51.729702 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:51.734223 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6598be485e94047be8d8a0279eb4595: Generated new TSK 0
I20260925 14:49:51.734437 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:51.738736 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--builtin_ntp_servers=127.0.98.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:49:51.827448 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:51.827687 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:51.827723 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:51.828959 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:51.829031 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:51.829073 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:51.830466 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:51.830735 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:51.831051 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:51.836156 9703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:51.836133 9701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:51.836120 9700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:51.836833 9695 server_base.cc:1034] running on GCE node
I20260925 14:49:51.837075 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:51.837327 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:51.838454 9695 hybrid_clock.cc:648] HybridClock initialized: now 1790347791838431 us; error 40 us; skew 500 ppm
I20260925 14:49:51.839980 9695 webserver.cc:460] Webserver started at http://127.0.98.1:43697/ using document root <none> and password file <none>
I20260925 14:49:51.840260 9695 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:51.840323 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:51.840452 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:51.841917 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "1184d6e37f02496d9a55bd83df549801"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "0d2ef2a79bf762e1950407f52e651493"
server_key_iv: "8da50246dc844499cc61bebfe8739b00"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.842375 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1184d6e37f02496d9a55bd83df549801"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "0d2ef2a79bf762e1950407f52e651493"
server_key_iv: "8da50246dc844499cc61bebfe8739b00"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.844291 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 14:49:51.845381 9709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.845557 9695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:49:51.845665 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "1184d6e37f02496d9a55bd83df549801"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "0d2ef2a79bf762e1950407f52e651493"
server_key_iv: "8da50246dc844499cc61bebfe8739b00"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.845741 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:51.865938 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:51.866264 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:51.866458 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:51.866714 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:51.867067 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:51.867101 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.867156 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:51.867182 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.872761 9695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45391
I20260925 14:49:51.872834 9822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45391 every 8 connection(s)
I20260925 14:49:51.873172 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:51.875862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9695
I20260925 14:49:51.875952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:51.876157 392 external_mini_cluster.cc:1428] Setting key 2704d88db1dd48cbbf2e2ddf044f3eb9
I20260925 14:49:51.877105 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--builtin_ntp_servers=127.0.98.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 14:49:51.879500 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.873997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41009 (local address 127.0.98.62:42815)
0925 14:49:51.874072 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:51.874075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:51.874340 (+ 265us) server_negotiation.cc:407] Connection header received
0925 14:49:51.874921 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:51.874922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:51.874939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:51.875017 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:51.875702 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.876505 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.877762 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.878099 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.878177 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.879235 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:51.879236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:51.879236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:51.879240 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:51.879254 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:51.879258 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:51.879304 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:51.879411 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:49:51.879443 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:49:51.879794 9825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.874007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42815 (local address 127.0.98.1:41009)
0925 14:49:51.874218 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:51.874252 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:51.874818 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:51.875071 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:51.875074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:51.875089 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:51.875573 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.875578 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.876606 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.876608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.877677 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.877680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.878229 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.878230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.878440 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.879067 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:51.879068 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:51.879109 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:51.879139 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:51.879325 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:51.879330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:51.879333 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:51.879409 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:49:51.879522 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260925 14:49:51.880441 9823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42815
I20260925 14:49:51.880558 9823 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:51.880780 9823 heartbeater.cc:511] Master 127.0.98.62:42815 requested a full tablet report, sending...
I20260925 14:49:51.881489 9635 ts_manager.cc:194] Registered new tserver with Master: 1184d6e37f02496d9a55bd83df549801 (127.0.98.1:45391)
I20260925 14:49:51.882148 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41009
W20260925 14:49:51.960235 9826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:51.960446 9826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:51.960469 9826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:51.961673 9826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:51.961719 9826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:51.961752 9826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:51.963156 9826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:51.963423 9826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:51.963757 9826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:51.969167 9831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:51.969189 9832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:51.969233 9826 server_base.cc:1034] running on GCE node
W20260925 14:49:51.969173 9834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:51.969537 9826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:51.969779 9826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:51.970938 9826 hybrid_clock.cc:648] HybridClock initialized: now 1790347791970927 us; error 30 us; skew 500 ppm
I20260925 14:49:51.972327 9826 webserver.cc:460] Webserver started at http://127.0.98.2:44393/ using document root <none> and password file <none>
I20260925 14:49:51.972595 9826 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:51.972666 9826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:51.972822 9826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:51.973908 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "63b8457e964c4163a3be6ab83e88482d"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "1037ab29ca5aef22c93bfab22069a61c"
server_key_iv: "f6f82183dd88ffc3d19fa99249842f44"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.974282 9826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "63b8457e964c4163a3be6ab83e88482d"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "1037ab29ca5aef22c93bfab22069a61c"
server_key_iv: "f6f82183dd88ffc3d19fa99249842f44"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.975612 9826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:51.976441 9840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.976672 9826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:51.976769 9826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "63b8457e964c4163a3be6ab83e88482d"
format_stamp: "Formatted at 2026-09-25 14:49:51 on dist-test-slave-z2s5"
server_key: "1037ab29ca5aef22c93bfab22069a61c"
server_key_iv: "f6f82183dd88ffc3d19fa99249842f44"
server_key_version: "encryptionkey@0"
I20260925 14:49:51.976853 9826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:51.987293 9826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:51.987610 9826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:51.987798 9826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:51.988037 9826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:51.988368 9826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:51.988426 9826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.988468 9826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:51.988497 9826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:51.994349 9826 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46711
I20260925 14:49:51.994446 9953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46711 every 8 connection(s)
I20260925 14:49:51.994774 9826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:52.000087 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.995557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60245 (local address 127.0.98.62:42815)
0925 14:49:51.995628 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:51.995631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:51.995992 (+ 361us) server_negotiation.cc:407] Connection header received
0925 14:49:51.996502 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:51.996503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:51.996514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:51.996584 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:51.997235 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.997706 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.998764 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.998962 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.999016 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.999809 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:51.999810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:51.999811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:51.999814 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:51.999827 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:51.999830 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:51.999895 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:52.000009 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 14:49:52.000030 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:49:52.000325 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:51.995558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42815 (local address 127.0.98.2:60245)
0925 14:49:51.995830 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:51.995867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:51.996402 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:51.996636 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:51.996639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:51.996647 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:51.997141 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.997145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.997831 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.997832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.998685 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:51.998688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:51.999047 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:51.999048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:51.999158 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:51.999666 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:51.999667 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:51.999695 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:51.999721 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:51.999923 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:51.999928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:51.999931 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:52.000004 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:49:52.000083 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
I20260925 14:49:52.000829 9954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42815
I20260925 14:49:52.000921 9954 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:52.001056 9954 heartbeater.cc:511] Master 127.0.98.62:42815 requested a full tablet report, sending...
I20260925 14:49:52.001430 9635 ts_manager.cc:194] Registered new tserver with Master: 63b8457e964c4163a3be6ab83e88482d (127.0.98.2:46711)
I20260925 14:49:52.001791 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60245
I20260925 14:49:52.002588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9826
I20260925 14:49:52.002655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:52.002813 392 external_mini_cluster.cc:1428] Setting key 3a1d8103e070c508e311d0980a438c36
I20260925 14:49:52.003620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--builtin_ntp_servers=127.0.98.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:49:52.082465 9957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:52.082660 9957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:52.082680 9957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:52.083761 9957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:52.083798 9957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:52.083822 9957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:52.085064 9957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:52.085242 9957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:52.085482 9957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:52.090786 9963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.090806 9965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.090888 9957 server_base.cc:1034] running on GCE node
W20260925 14:49:52.090826 9962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.091317 9957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:52.091583 9957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:52.092746 9957 hybrid_clock.cc:648] HybridClock initialized: now 1790347792092730 us; error 43 us; skew 500 ppm
I20260925 14:49:52.094238 9957 webserver.cc:460] Webserver started at http://127.0.98.3:35095/ using document root <none> and password file <none>
I20260925 14:49:52.094530 9957 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:52.094604 9957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:52.094753 9957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:52.095803 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb2521a95c6144c4adbeedd85003f6ba"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "4099fa4600dfdbbf15396e84a17decef"
server_key_iv: "cc8b1801eb57d4aa8cca935562ad4273"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.096166 9957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb2521a95c6144c4adbeedd85003f6ba"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "4099fa4600dfdbbf15396e84a17decef"
server_key_iv: "cc8b1801eb57d4aa8cca935562ad4273"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.097487 9957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:52.098256 9971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.098493 9957 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:52.098600 9957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "bb2521a95c6144c4adbeedd85003f6ba"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "4099fa4600dfdbbf15396e84a17decef"
server_key_iv: "cc8b1801eb57d4aa8cca935562ad4273"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.098688 9957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:52.108144 9957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:52.108464 9957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:52.108613 9957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:52.108843 9957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:52.109184 9957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:52.109239 9957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.109282 9957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:52.109308 9957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.115330 9957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44487
I20260925 14:49:52.115377 10084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44487 every 8 connection(s)
I20260925 14:49:52.115718 9957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:52.118135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 9957
I20260925 14:49:52.118211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:52.118507 392 external_mini_cluster.cc:1428] Setting key 6ab3d06c2af5f1953f1344ae8b57c6c5
I20260925 14:49:52.121429 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.116644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57225 (local address 127.0.98.62:42815)
0925 14:49:52.116690 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.116692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.116974 (+ 282us) server_negotiation.cc:407] Connection header received
0925 14:49:52.117528 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.117529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.117542 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.117600 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.118285 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:52.118856 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:52.119988 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:52.120192 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:52.120281 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:52.121136 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:52.121138 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:52.121138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:52.121143 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:52.121156 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:52.121160 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:52.121231 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:52.121327 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 14:49:52.121353 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:52.121685 10087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.116638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42815 (local address 127.0.98.3:57225)
0925 14:49:52.116843 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:52.116879 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:52.117407 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:52.117678 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:52.117682 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:52.117691 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:52.118207 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:52.118213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:52.118976 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:52.118978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:52.119896 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:52.119899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:52.120290 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:52.120291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:52.120426 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:52.120997 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:52.120997 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:52.121024 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:52.121047 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:52.121231 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:52.121237 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:52.121240 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:52.121331 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:49:52.121441 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260925 14:49:52.122170 10085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42815
I20260925 14:49:52.122267 10085 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:52.122459 10085 heartbeater.cc:511] Master 127.0.98.62:42815 requested a full tablet report, sending...
I20260925 14:49:52.122838 9635 ts_manager.cc:194] Registered new tserver with Master: bb2521a95c6144c4adbeedd85003f6ba (127.0.98.3:44487)
I20260925 14:49:52.123207 9635 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:57225
I20260925 14:49:52.130307 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:49:52.132414 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:52.131695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.0.98.62:42815)
0925 14:49:52.131836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.131838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.131999 (+ 161us) server_negotiation.cc:407] Connection header received
0925 14:49:52.132086 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.132087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.132098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.132182 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.132322 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33696: BlockingRecv error: recv got EOF from 127.0.0.1:33696 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
I20260925 14:49:52.132467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9695
I20260925 14:49:52.136904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9826
I20260925 14:49:52.141010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9957
I20260925 14:49:52.145215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 9604
2026-09-25T14:49:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (581 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:49:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:52Z Disabled control of system clock
I20260925 14:49:52.171672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35047
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:49:52.251902 10096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:52.252113 10096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:52.252134 10096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:52.253214 10096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:52.253250 10096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:52.253263 10096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:52.253276 10096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:52.253288 10096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:52.254571 10096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:52.254763 10096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:52.255009 10096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:52.260025 10102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.260017 10104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.260331 10096 server_base.cc:1034] running on GCE node
W20260925 14:49:52.260017 10101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.260670 10096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:52.260923 10096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:52.262068 10096 hybrid_clock.cc:648] HybridClock initialized: now 1790347792262046 us; error 40 us; skew 500 ppm
I20260925 14:49:52.263365 10096 webserver.cc:460] Webserver started at http://127.0.98.62:37663/ using document root <none> and password file <none>
I20260925 14:49:52.263621 10096 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:52.263669 10096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:52.263847 10096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:52.264956 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "3cecd57d8658453a9e44f3d25965446f"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "f177b502d6cdd646e042434048d01eee"
server_key_iv: "9160573bd20c63a183af3606b51d403d"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.265318 10096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cecd57d8658453a9e44f3d25965446f"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "f177b502d6cdd646e042434048d01eee"
server_key_iv: "9160573bd20c63a183af3606b51d403d"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.266634 10096 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:49:52.267316 10110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.267477 10096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:49:52.267577 10096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "3cecd57d8658453a9e44f3d25965446f"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "f177b502d6cdd646e042434048d01eee"
server_key_iv: "9160573bd20c63a183af3606b51d403d"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.267657 10096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:52.270943 10096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:52.271188 10096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:52.271337 10096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:52.274948 10096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35047
I20260925 14:49:52.275012 10162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35047 every 8 connection(s)
I20260925 14:49:52.275323 10096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:52.275812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10096
I20260925 14:49:52.275931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:52.276125 392 external_mini_cluster.cc:1428] Setting key db5d9f28fce7fc6cca68696a62fa34c4
I20260925 14:49:52.276145 10163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:52.279094 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.276837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.98.62:35047)
0925 14:49:52.277026 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.277044 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.277076 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:49:52.277625 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.277635 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.277688 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.277854 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.278575 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:52.278579 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:52.278579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:52.278584 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:52.278632 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:52.278645 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:52.278726 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:52.278859 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 14:49:52.278913 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260925 14:49:52.279517 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f: Bootstrap starting.
I20260925 14:49:52.280095 10163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:52.280380 10163 log.cc:824] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:52.281082 10163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f: No bootstrap required, opened a new log
I20260925 14:49:52.282373 10163 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cecd57d8658453a9e44f3d25965446f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35047 } }
I20260925 14:49:52.282593 10163 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:52.282644 10163 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cecd57d8658453a9e44f3d25965446f, State: Initialized, Role: FOLLOWER
I20260925 14:49:52.282773 10163 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cecd57d8658453a9e44f3d25965446f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35047 } }
I20260925 14:49:52.282866 10163 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:52.282909 10163 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:52.282958 10163 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:52.283483 10163 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cecd57d8658453a9e44f3d25965446f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35047 } }
I20260925 14:49:52.283599 10163 leader_election.cc:302] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cecd57d8658453a9e44f3d25965446f; no voters:
I20260925 14:49:52.283761 10163 leader_election.cc:288] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:52.283834 10168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:52.283986 10168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [term 1 LEADER]: Becoming Leader. State: Replica: 3cecd57d8658453a9e44f3d25965446f, State: Running, Role: LEADER
I20260925 14:49:52.284051 10163 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:52.284076 10168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cecd57d8658453a9e44f3d25965446f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35047 } }
I20260925 14:49:52.284497 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cecd57d8658453a9e44f3d25965446f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cecd57d8658453a9e44f3d25965446f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35047 } } }
I20260925 14:49:52.284528 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cecd57d8658453a9e44f3d25965446f. Latest consensus state: current_term: 1 leader_uuid: "3cecd57d8658453a9e44f3d25965446f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cecd57d8658453a9e44f3d25965446f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35047 } } }
I20260925 14:49:52.284653 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:52.284695 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:52.285099 10177 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:52.285568 10177 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:52.287087 10177 catalog_manager.cc:1347] Generated new cluster ID: b35d794e418043868faa4e703881c2c0
I20260925 14:49:52.287153 10177 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:52.307621 10177 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:52.307827 10177 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:52.313196 10177 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cecd57d8658453a9e44f3d25965446f: Generated new TSK 0
I20260925 14:49:52.313391 10177 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:49:52.315490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35047
--builtin_ntp_servers=127.0.98.20:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:49:52.412036 10187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:52.412262 10187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:52.412302 10187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:52.413538 10187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:52.413594 10187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:52.413642 10187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:52.415160 10187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:52.415426 10187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:52.415704 10187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:52.421753 10193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.421753 10192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.422501 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.422533 10187 server_base.cc:1034] running on GCE node
I20260925 14:49:52.422791 10187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:52.423062 10187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:52.424233 10187 hybrid_clock.cc:648] HybridClock initialized: now 1790347792424223 us; error 26 us; skew 500 ppm
I20260925 14:49:52.425642 10187 webserver.cc:460] Webserver started at http://127.0.98.1:32785/ using document root <none> and password file <none>
I20260925 14:49:52.425881 10187 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:52.425958 10187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:52.426095 10187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:52.427316 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "faffec1d796242e19e1ac352649f0ae5"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "68f18fb95ef06ea6fa0553592d34cdb8"
server_key_iv: "0fd2db8c5f231f0596d713861930ed1f"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.427699 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "faffec1d796242e19e1ac352649f0ae5"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "68f18fb95ef06ea6fa0553592d34cdb8"
server_key_iv: "0fd2db8c5f231f0596d713861930ed1f"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.429071 10187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:49:52.429827 10201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.430073 10187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:49:52.430172 10187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "faffec1d796242e19e1ac352649f0ae5"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "68f18fb95ef06ea6fa0553592d34cdb8"
server_key_iv: "0fd2db8c5f231f0596d713861930ed1f"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.430245 10187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:52.435146 10187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:52.435447 10187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:52.435567 10187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:52.435751 10187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:52.436058 10187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:52.436096 10187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.436125 10187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:52.436143 10187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.442930 10187 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35211
I20260925 14:49:52.442962 10314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35211 every 8 connection(s)
I20260925 14:49:52.443351 10187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:52.446290 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.444099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44681 (local address 127.0.98.62:35047)
0925 14:49:52.444168 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.444170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.444365 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:49:52.444819 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.444820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.444824 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.444901 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.446063 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:52.446065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:52.446065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:52.446069 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:52.446082 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:52.446086 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:52.446138 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:52.446223 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 14:49:52.446235 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:49:52.446521 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.444032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35047 (local address 127.0.98.1:44681)
0925 14:49:52.444229 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:52.444269 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:52.444737 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:52.444972 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:52.444975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:52.444984 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:52.445915 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:52.445916 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:52.445944 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:52.445971 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:52.446151 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:52.446155 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:52.446157 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:52.446215 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:49:52.446264 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260925 14:49:52.447016 10315 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35047
I20260925 14:49:52.447088 10315 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:52.447194 10315 heartbeater.cc:511] Master 127.0.98.62:35047 requested a full tablet report, sending...
I20260925 14:49:52.447638 10127 ts_manager.cc:194] Registered new tserver with Master: faffec1d796242e19e1ac352649f0ae5 (127.0.98.1:35211)
I20260925 14:49:52.451051 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10187
I20260925 14:49:52.451128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:52.451315 392 external_mini_cluster.cc:1428] Setting key 42dba59374da448cd02f7973071ee792
I20260925 14:49:52.452229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35047
--builtin_ntp_servers=127.0.98.20:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:49:52.531651 10318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:52.531864 10318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:52.531888 10318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:52.533075 10318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:52.533116 10318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:52.533142 10318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:52.534742 10318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:52.535006 10318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:52.535332 10318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:49:52.540506 10318 server_base.cc:1034] running on GCE node
W20260925 14:49:52.540453 10326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.540472 10324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.540439 10323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.540925 10318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:52.541134 10318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:52.542301 10318 hybrid_clock.cc:648] HybridClock initialized: now 1790347792542295 us; error 37 us; skew 500 ppm
I20260925 14:49:52.543596 10318 webserver.cc:460] Webserver started at http://127.0.98.2:45099/ using document root <none> and password file <none>
I20260925 14:49:52.543834 10318 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:52.543920 10318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:52.544065 10318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:52.545143 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8d8c36814f34d7b9de80060ae4556a4"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "144259bce9b9d240f90ed81cdcfaa644"
server_key_iv: "57b10a0c0e131a9b7b4772b7ef0ecac5"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.545507 10318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8d8c36814f34d7b9de80060ae4556a4"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "144259bce9b9d240f90ed81cdcfaa644"
server_key_iv: "57b10a0c0e131a9b7b4772b7ef0ecac5"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.546919 10318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:52.547643 10332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.547843 10318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.547960 10318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "c8d8c36814f34d7b9de80060ae4556a4"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "144259bce9b9d240f90ed81cdcfaa644"
server_key_iv: "57b10a0c0e131a9b7b4772b7ef0ecac5"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.548038 10318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:52.551354 10318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:52.551611 10318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:52.551755 10318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:52.551970 10318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:52.552268 10318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:52.552325 10318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.552363 10318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:52.552389 10318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.558096 10318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44223
I20260925 14:49:52.558204 10445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44223 every 8 connection(s)
I20260925 14:49:52.558569 10318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:49:52.561504 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.559375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37111 (local address 127.0.98.62:35047)
0925 14:49:52.559420 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.559422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.559663 (+ 241us) server_negotiation.cc:407] Connection header received
0925 14:49:52.560149 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.560150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.560156 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.560208 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.561308 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:52.561309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:52.561310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:52.561313 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:52.561326 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:52.561330 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:52.561376 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:52.561437 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:49:52.561450 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:49:52.561687 10448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.559351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35047 (local address 127.0.98.2:37111)
0925 14:49:52.559534 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:52.559568 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:52.560058 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:52.560263 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:52.560266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:52.560274 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:52.561168 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:52.561169 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:52.561196 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:52.561224 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:52.561374 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:52.561376 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:52.561378 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:52.561430 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:49:52.561476 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260925 14:49:52.562139 10446 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35047
I20260925 14:49:52.562218 10446 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:52.562434 10446 heartbeater.cc:511] Master 127.0.98.62:35047 requested a full tablet report, sending...
I20260925 14:49:52.562840 10127 ts_manager.cc:194] Registered new tserver with Master: c8d8c36814f34d7b9de80060ae4556a4 (127.0.98.2:44223)
I20260925 14:49:52.566695 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10318
I20260925 14:49:52.566771 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:52.566978 392 external_mini_cluster.cc:1428] Setting key 3e687396c393f86ad324f236f6d08c6e
I20260925 14:49:52.567863 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35047
--builtin_ntp_servers=127.0.98.20:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:49:52.647498 10449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:52.647709 10449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:52.647732 10449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:52.648906 10449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:52.648948 10449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:52.648976 10449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:52.650298 10449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:52.650545 10449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:52.650794 10449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:52.655790 10454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.655781 10455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:52.655783 10457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:52.656105 10449 server_base.cc:1034] running on GCE node
I20260925 14:49:52.656337 10449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:52.656600 10449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:52.657764 10449 hybrid_clock.cc:648] HybridClock initialized: now 1790347792657759 us; error 34 us; skew 500 ppm
I20260925 14:49:52.659117 10449 webserver.cc:460] Webserver started at http://127.0.98.3:36209/ using document root <none> and password file <none>
I20260925 14:49:52.659376 10449 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:52.659448 10449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:52.659600 10449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:52.660682 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "008f793e7b464ae99962e25e46a14c87"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "8102027603a5a3b78361fa3e0e9f44a7"
server_key_iv: "355341eeb4fc8bd190a862f131f70cae"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.661068 10449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "008f793e7b464ae99962e25e46a14c87"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "8102027603a5a3b78361fa3e0e9f44a7"
server_key_iv: "355341eeb4fc8bd190a862f131f70cae"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.662470 10449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:52.663250 10463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.663429 10449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:52.663530 10449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "008f793e7b464ae99962e25e46a14c87"
format_stamp: "Formatted at 2026-09-25 14:49:52 on dist-test-slave-z2s5"
server_key: "8102027603a5a3b78361fa3e0e9f44a7"
server_key_iv: "355341eeb4fc8bd190a862f131f70cae"
server_key_version: "encryptionkey@0"
I20260925 14:49:52.663617 10449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:52.666917 10449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:52.667181 10449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:52.667327 10449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:52.667544 10449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:52.667848 10449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:52.667905 10449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.667943 10449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:52.667972 10449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:52.673772 10449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40001
I20260925 14:49:52.673839 10576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40001 every 8 connection(s)
I20260925 14:49:52.674188 10449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:49:52.677160 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.675004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43399 (local address 127.0.98.62:35047)
0925 14:49:52.675088 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.675091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.675337 (+ 246us) server_negotiation.cc:407] Connection header received
0925 14:49:52.675758 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.675759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.675764 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.675826 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.676965 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:52.676966 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:52.676966 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:49:52.676970 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:52.676984 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:49:52.676988 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:52.677034 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:52.677095 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 14:49:52.677107 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:49:52.677361 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:52.675001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35047 (local address 127.0.98.3:43399)
0925 14:49:52.675189 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:52.675230 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:52.675672 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:52.675885 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:52.675888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:52.675896 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:52.676773 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:49:52.676773 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:52.676801 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:49:52.676829 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:52.677032 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:52.677034 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:52.677036 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:49:52.677089 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:49:52.677136 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260925 14:49:52.677865 10577 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35047
I20260925 14:49:52.677954 10577 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:52.678063 10577 heartbeater.cc:511] Master 127.0.98.62:35047 requested a full tablet report, sending...
I20260925 14:49:52.678459 10127 ts_manager.cc:194] Registered new tserver with Master: 008f793e7b464ae99962e25e46a14c87 (127.0.98.3:40001)
I20260925 14:49:52.682590 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10449
I20260925 14:49:52.682667 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:52.682895 392 external_mini_cluster.cc:1428] Setting key ab28285c298f899da94bd01424b56e8d
I20260925 14:49:52.683780 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:49:52.685572 10166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:49:52.685188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.0.98.62:35047)
0925 14:49:52.685226 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:52.685228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:52.685333 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:49:52.685387 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:52.685387 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:52.685392 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:52.685443 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:52.685528 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38258: BlockingRecv error: recv got EOF from 127.0.0.1:38258 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:49:52.685652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10187
I20260925 14:49:52.690488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10318
I20260925 14:49:52.694505 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10449
I20260925 14:49:52.698673 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10096
2026-09-25T14:49:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (552 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:52 dist-test-slave-z2s5 krb5kdc[10587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:52 dist-test-slave-z2s5 krb5kdc[10587](info): set up 2 sockets
Sep 25 14:49:52 dist-test-slave-z2s5 krb5kdc[10587](info): commencing operation
krb5kdc: starting...
W20260925 14:49:53.739428 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:53 dist-test-slave-z2s5 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:49:53.860867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:53.940595 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:53.940781 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:53.940810 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:53.941972 10600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:49:53.942013 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:53.942027 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:53.942039 10600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:49:53.942102 10600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:49:53.943543 10600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:53.943784 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:53.944023 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:53.949076 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:53.949112 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:53.949092 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:53.949210 10600 server_base.cc:1034] running on GCE node
I20260925 14:49:53.949596 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:53.949841 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:53.950984 10600 hybrid_clock.cc:648] HybridClock initialized: now 1790347793950958 us; error 42 us; skew 500 ppm
Sep 25 14:49:53 dist-test-slave-z2s5 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:53.953361 10600 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:49:53.953801 10600 webserver.cc:460] Webserver started at http://127.0.98.62:41691/ using document root <none> and password file <none>
I20260925 14:49:53.954043 10600 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:53.954116 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:53.954267 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:53.955468 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6"
format_stamp: "Formatted at 2026-09-25 14:49:53 on dist-test-slave-z2s5"
server_key: "dd7aeda87e4e062f89e97502a0a93f3e"
server_key_iv: "93cfb25b941978b4cc06b2a6f6de3eb9"
server_key_version: "encryptionkey@0"
I20260925 14:49:53.955816 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6"
format_stamp: "Formatted at 2026-09-25 14:49:53 on dist-test-slave-z2s5"
server_key: "dd7aeda87e4e062f89e97502a0a93f3e"
server_key_iv: "93cfb25b941978b4cc06b2a6f6de3eb9"
server_key_version: "encryptionkey@0"
I20260925 14:49:53.957194 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:53.957911 10615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:53.958076 10600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:49:53.958178 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6"
format_stamp: "Formatted at 2026-09-25 14:49:53 on dist-test-slave-z2s5"
server_key: "dd7aeda87e4e062f89e97502a0a93f3e"
server_key_iv: "93cfb25b941978b4cc06b2a6f6de3eb9"
server_key_version: "encryptionkey@0"
I20260925 14:49:53.958258 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:53.996408 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:53.998495 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:53.998699 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:54.002872 10600 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39381
I20260925 14:49:54.002904 10667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39381 every 8 connection(s)
I20260925 14:49:54.003328 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:49:54.004343 10668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:54.005784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10600
I20260925 14:49:54.005934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:49:54.006121 392 external_mini_cluster.cc:1428] Setting key f750c78254642c05a3c35f288a831514
I20260925 14:49:54.007117 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6: Bootstrap starting.
I20260925 14:49:54.007802 10668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:54.008140 10668 log.cc:824] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:54.009084 10668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6: No bootstrap required, opened a new log
I20260925 14:49:54.011340 10668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260925 14:49:54.011531 10668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:54.011574 10668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7270d3cfdc44a8a26af27a1bddeaa6, State: Initialized, Role: FOLLOWER
I20260925 14:49:54.011705 10668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260925 14:49:54.011798 10668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:49:54.011862 10668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:49:54.011902 10668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:54.012723 10668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260925 14:49:54.012845 10668 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7270d3cfdc44a8a26af27a1bddeaa6; no voters:
I20260925 14:49:54.013016 10668 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:49:54.013077 10673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:54.013243 10673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7270d3cfdc44a8a26af27a1bddeaa6, State: Running, Role: LEADER
I20260925 14:49:54.013296 10668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:49:54.013329 10673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } }
I20260925 14:49:54.013629 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } } }
I20260925 14:49:54.013746 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [sys.catalog]: This master's current role is: LEADER
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347793, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:54.013628 10675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7270d3cfdc44a8a26af27a1bddeaa6. Latest consensus state: current_term: 1 leader_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7270d3cfdc44a8a26af27a1bddeaa6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39381 } } }
I20260925 14:49:54.013870 10675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6 [sys.catalog]: This master's current role is: LEADER
I20260925 14:49:54.014055 10680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:49:54.014545 10680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:49:54.016319 10680 catalog_manager.cc:1347] Generated new cluster ID: 46a8aa91d0574a9d9fc5041cde27b0b8
I20260925 14:49:54.016386 10680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:49:54.018186 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.006693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.0.98.62:39381)
0925 14:49:54.006891 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.006901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.006934 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:49:54.007482 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.007493 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.007568 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.007716 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:54.008519 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.009665 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.010787 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.011223 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.011335 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.014498 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:54.014502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:54.014503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:54.014517 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:54.016654 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.017034 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.017037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.017037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.017134 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.017348 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.017349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.017349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.017591 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:54.017756 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.017867 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.018016 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260925 14:49:54.033200 10680 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:49:54.033949 10680 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:49:54.036701 10680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6: Generated new TSK 0
I20260925 14:49:54.036863 10680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:49:54.062005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39381
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:54.142127 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:54.142319 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:54.142341 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:54.143520 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:54.143567 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:54.143594 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:49:54.145097 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:54.145357 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:54.145558 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:54.150763 10700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:54.150807 10702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:54.150960 10694 server_base.cc:1034] running on GCE node
W20260925 14:49:54.150780 10699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:54.151297 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:54.151507 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:54.152644 10694 hybrid_clock.cc:648] HybridClock initialized: now 1790347794152624 us; error 37 us; skew 500 ppm
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:54.155102 10694 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:49:54.155566 10694 webserver.cc:460] Webserver started at http://127.0.98.1:46573/ using document root <none> and password file <none>
I20260925 14:49:54.155810 10694 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:54.155856 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:54.155999 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:54.157063 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "e95d1ea115da455999be13b6b12a1d25"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "8143640d2d28e388acdb5eb4fb38a1fe"
server_key_iv: "67e1aae5a8f73a6d1124801a9248b349"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.157420 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e95d1ea115da455999be13b6b12a1d25"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "8143640d2d28e388acdb5eb4fb38a1fe"
server_key_iv: "67e1aae5a8f73a6d1124801a9248b349"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.158798 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:49:54.159531 10709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.159703 10694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.159801 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "e95d1ea115da455999be13b6b12a1d25"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "8143640d2d28e388acdb5eb4fb38a1fe"
server_key_iv: "67e1aae5a8f73a6d1124801a9248b349"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.159888 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:54.185936 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:54.188006 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:54.188184 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:54.188444 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:54.188761 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:54.188817 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.188853 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:54.188881 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.194537 10694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37013
I20260925 14:49:54.194587 10822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37013 every 8 connection(s)
I20260925 14:49:54.194898 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:49:54.196907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10694
I20260925 14:49:54.197055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:49:54.197263 392 external_mini_cluster.cc:1428] Setting key ab694e270702c9a286f1749ed1128bd4
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:49:54.205605 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.195637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57809 (local address 127.0.98.62:39381)
0925 14:49:54.195696 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.195698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.195923 (+ 225us) server_negotiation.cc:407] Connection header received
0925 14:49:54.196480 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.196482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.196494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.196548 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:54.197903 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.198460 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.199828 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.200054 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.200145 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.202703 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:54.202705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:54.202705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:54.202713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:54.204116 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.204588 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.204590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.204590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.204652 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.204909 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.204917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.204918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.205038 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:54.205121 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.205466 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.205539 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:49:54.205813 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.195619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39381 (local address 127.0.98.1:57809)
0925 14:49:54.195772 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.195810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.196369 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.196636 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.196639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.197527 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:54.197769 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.197774 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.198545 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.198547 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.199722 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.199724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.200153 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.200154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.200286 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.200819 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:54.200827 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:54.202596 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:54.204221 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:54.204228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:54.204231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:54.204478 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:54.204728 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:54.204729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:54.204729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:54.204841 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:54.205132 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:54.205140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:54.205317 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.205444 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.205593 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260925 14:49:54.206521 10823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39381
I20260925 14:49:54.206629 10823 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:54.206807 10823 heartbeater.cc:511] Master 127.0.98.62:39381 requested a full tablet report, sending...
I20260925 14:49:54.207406 10632 ts_manager.cc:194] Registered new tserver with Master: e95d1ea115da455999be13b6b12a1d25 (127.0.98.1:37013)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 14:49:54.208181 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:49:54.225152 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39381
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:54.308254 10828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:54.308472 10828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:54.308496 10828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:54.309593 10828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:54.309629 10828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:54.309654 10828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:49:54.311671 10828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:54.311865 10828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:54.312037 10828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:54.317178 10834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:54.317178 10833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:54.317407 10828 server_base.cc:1034] running on GCE node
W20260925 14:49:54.317162 10836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:54.317677 10828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:54.317873 10828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:54.319003 10828 hybrid_clock.cc:648] HybridClock initialized: now 1790347794318993 us; error 28 us; skew 500 ppm
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:54.321482 10828 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:49:54.321933 10828 webserver.cc:460] Webserver started at http://127.0.98.2:38041/ using document root <none> and password file <none>
I20260925 14:49:54.322121 10828 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:54.322161 10828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:54.322248 10828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:54.323309 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "05dd1448cde74463a87d07b082b6b7d7"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "ba7187493205becf9b9c3d37de7c942f"
server_key_iv: "1c51fae76e35e43db8b4d50ef0802414"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.323618 10828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "05dd1448cde74463a87d07b082b6b7d7"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "ba7187493205becf9b9c3d37de7c942f"
server_key_iv: "1c51fae76e35e43db8b4d50ef0802414"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.324887 10828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:49:54.325659 10843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.325867 10828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:54.325956 10828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "05dd1448cde74463a87d07b082b6b7d7"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "ba7187493205becf9b9c3d37de7c942f"
server_key_iv: "1c51fae76e35e43db8b4d50ef0802414"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.326006 10828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:54.344879 10828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:54.347301 10828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:54.347422 10828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:54.347613 10828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:54.347870 10828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:54.347901 10828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.347922 10828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:54.347935 10828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.353709 10828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43887
I20260925 14:49:54.353756 10956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43887 every 8 connection(s)
I20260925 14:49:54.354071 10828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:54.360361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10828
I20260925 14:49:54.360479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:49:54.360674 392 external_mini_cluster.cc:1428] Setting key 905bad63182f94e5b1b6171df456be05
I20260925 14:49:54.364184 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.354868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40105 (local address 127.0.98.62:39381)
0925 14:49:54.354985 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.354987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.355164 (+ 177us) server_negotiation.cc:407] Connection header received
0925 14:49:54.355725 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.355727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.355739 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.355791 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:54.356767 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.357262 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.358267 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.358519 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.358602 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.360945 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:54.360946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:54.360946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:54.360953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:54.362540 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.363077 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.363079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.363079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.363143 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.363421 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.363422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.363422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.363541 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:54.363620 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.363949 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.364039 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:49:54.364262 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.354857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39381 (local address 127.0.98.2:40105)
0925 14:49:54.355027 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.355078 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.355604 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.355880 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.355886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.356455 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:54.356650 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.356655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.357360 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.357361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.358165 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.358167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.358612 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.358614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.358742 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.359192 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:54.359201 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:54.360748 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:54.362620 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:54.362626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:54.362629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:54.362987 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:54.363211 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:54.363212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:54.363212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:54.363333 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:54.363611 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:54.363617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:54.363763 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.363887 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.364078 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260925 14:49:54.364791 10957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39381
I20260925 14:49:54.364920 10957 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:54.365103 10957 heartbeater.cc:511] Master 127.0.98.62:39381 requested a full tablet report, sending...
I20260925 14:49:54.365506 10632 ts_manager.cc:194] Registered new tserver with Master: 05dd1448cde74463a87d07b082b6b7d7 (127.0.98.2:43887)
I20260925 14:49:54.366065 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40105
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:49:54.385650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39381
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:49:54.471695 10962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:49:54.471925 10962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:49:54.471966 10962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:49:54.473102 10962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:49:54.473172 10962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:49:54.473219 10962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:49:54.474752 10962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:49:54.475025 10962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:49:54.475250 10962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:49:54.480187 10970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:49:54.480175 10968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:54.480273 10962 server_base.cc:1034] running on GCE node
W20260925 14:49:54.480175 10967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:49:54.480614 10962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:49:54.480854 10962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:49:54.482028 10962 hybrid_clock.cc:648] HybridClock initialized: now 1790347794482029 us; error 41 us; skew 500 ppm
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:49:54.484550 10962 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:49:54.485002 10962 webserver.cc:460] Webserver started at http://127.0.98.3:45643/ using document root <none> and password file <none>
I20260925 14:49:54.485188 10962 fs_manager.cc:359] Metadata directory not provided
I20260925 14:49:54.485227 10962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:49:54.485316 10962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:49:54.486292 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "45acaadd398f4136ab44718e997301a3"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "3d40e0ad461f26049da8af13cdb47904"
server_key_iv: "314c0db726e49c42d4ef0135fedd8db0"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.486673 10962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "45acaadd398f4136ab44718e997301a3"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "3d40e0ad461f26049da8af13cdb47904"
server_key_iv: "314c0db726e49c42d4ef0135fedd8db0"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.487874 10962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:49:54.488523 10977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.488684 10962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.488770 10962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "45acaadd398f4136ab44718e997301a3"
format_stamp: "Formatted at 2026-09-25 14:49:54 on dist-test-slave-z2s5"
server_key: "3d40e0ad461f26049da8af13cdb47904"
server_key_iv: "314c0db726e49c42d4ef0135fedd8db0"
server_key_version: "encryptionkey@0"
I20260925 14:49:54.488817 10962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:49:54.504199 10962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:49:54.506577 10962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:49:54.506692 10962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:49:54.506870 10962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:49:54.507215 10962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:49:54.507251 10962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.507273 10962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:49:54.507323 10962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:49:54.513062 10962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46003
I20260925 14:49:54.513144 11090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46003 every 8 connection(s)
I20260925 14:49:54.513502 10962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[10587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:49:54.522888 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.514263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58283 (local address 127.0.98.62:39381)
0925 14:49:54.514303 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.514305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.514543 (+ 238us) server_negotiation.cc:407] Connection header received
0925 14:49:54.515149 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.515151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.515162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.515226 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:54.516138 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.516620 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.517671 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.517913 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.517987 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.520217 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:54.520218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:54.520218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:54.520224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:54.521645 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.522022 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.522023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.522023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.522073 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.522290 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.522291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.522291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.522459 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:54.522514 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.522765 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.522823 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:49:54.523067 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.514222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39381 (local address 127.0.98.3:58283)
0925 14:49:54.514410 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.514454 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.515021 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.515291 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.515294 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.515862 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:49:54.516059 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.516064 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.516701 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.516702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.517569 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.517571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.517999 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.518000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.518147 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.518786 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:49:54.518795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:49:54.520131 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:49:54.521716 (+ 1585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:54.521721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:54.521722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:54.521955 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:54.522132 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:49:54.522133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:49:54.522133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:49:54.522227 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:49:54.522531 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:49:54.522535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:49:54.522651 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.522740 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.522855 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":110,"threads_started":1}
I20260925 14:49:54.523710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 10962
I20260925 14:49:54.523718 11091 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39381
I20260925 14:49:54.523797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:49:54.523836 11091 heartbeater.cc:464] Registering TS with master...
I20260925 14:49:54.524008 392 external_mini_cluster.cc:1428] Setting key 176aca876c350c2eb7828539e79e532e
I20260925 14:49:54.524022 11091 heartbeater.cc:511] Master 127.0.98.62:39381 requested a full tablet report, sending...
I20260925 14:49:54.524488 10632 ts_manager.cc:194] Registered new tserver with Master: 45acaadd398f4136ab44718e997301a3 (127.0.98.3:46003)
I20260925 14:49:54.524894 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:49:54.524967 10632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58283
I20260925 14:49:54.531780 10671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.526238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.98.62:39381)
0925 14:49:54.526276 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.526277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.526379 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:49:54.526488 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.526489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.526498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.526550 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:49:54.527053 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.527501 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.528170 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.528337 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.528385 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.529445 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:49:54.529446 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:49:54.529446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:49:54.529452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:49:54.530868 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.531168 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.531169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.531169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.531213 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:49:54.531361 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:49:54.531361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:49:54.531361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:49:54.531485 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:49:54.531554 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.531640 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.531694 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:49:54.533283 10632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:49:54.534937 10632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:49:54.541419 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.537086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37013 (local address 127.0.0.1:46270)
0925 14:49:54.537241 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.537263 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.537324 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.537988 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.537989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.537991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:54.538032 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.538033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.539358 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.539359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.540507 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.540509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.541167 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.541168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.541228 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.541232 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.541285 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.541348 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":19,"thread_start_us":60,"threads_started":1}
I20260925 14:49:54.541440 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.537422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.0.98.3:46003)
0925 14:49:54.537603 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.537608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.537971 (+ 363us) server_negotiation.cc:407] Connection header received
0925 14:49:54.538015 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.538018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.538034 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.538103 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:54.538229 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.538980 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.540467 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.541016 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.541073 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.541143 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.541293 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.541341 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260925 14:49:54.541682 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.537279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46003 (local address 127.0.0.1:48906)
0925 14:49:54.537842 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.537851 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.537903 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.538111 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.538112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.538114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:54.538176 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.538177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.539089 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.539090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.540367 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.540370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.541101 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.541102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.541189 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.541203 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.541274 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.541641 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":40,"threads_started":1}
I20260925 14:49:54.541467 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.537201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46270 (local address 127.0.98.1:37013)
0925 14:49:54.537647 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.537650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.537660 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:54.537721 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.537723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.537739 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.538106 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:54.538177 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.539238 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.540577 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.541046 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.541145 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.541207 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.541337 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.541383 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":81,"threads_started":1}
I20260925 14:49:54.542461 11025 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe95403b5fa45259fa205e139c6c99a (DEFAULT_TABLE table=test-table [id=deca427c1b1744558c49d159b24dcd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:54.542555 10757 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe95403b5fa45259fa205e139c6c99a (DEFAULT_TABLE table=test-table [id=deca427c1b1744558c49d159b24dcd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:54.542829 11025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe95403b5fa45259fa205e139c6c99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:54.542829 10757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe95403b5fa45259fa205e139c6c99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:54.546170 11106 tablet_bootstrap.cc:492] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Bootstrap starting.
I20260925 14:49:54.546170 11105 tablet_bootstrap.cc:492] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25: Bootstrap starting.
I20260925 14:49:54.546603 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.537086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43887 (local address 127.0.0.1:45808)
0925 14:49:54.537218 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.537262 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.537326 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.537799 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.537802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.537806 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:54.537880 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.537881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.539314 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.539314 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.540370 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.540371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.542143 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.542144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.546367 (+ 4223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.546372 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.546483 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.546534 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"spinlock_wait_cycles":768,"thread_start_us":62,"threads_started":1}
I20260925 14:49:54.546648 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.537461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.0.98.2:43887)
0925 14:49:54.537615 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.537619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.537629 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:49:54.537667 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.537674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.537686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.537747 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:54.537981 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.539194 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.541183 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.541912 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.542262 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.542338 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.546510 (+ 4172us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.546564 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260925 14:49:54.546955 11105 tablet_bootstrap.cc:654] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:54.547156 11106 tablet_bootstrap.cc:654] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:54.547354 10891 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe95403b5fa45259fa205e139c6c99a (DEFAULT_TABLE table=test-table [id=deca427c1b1744558c49d159b24dcd56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:49:54.547575 11105 log.cc:824] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:54.547713 10891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe95403b5fa45259fa205e139c6c99a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:49:54.547716 11106 log.cc:824] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:54.548460 11105 tablet_bootstrap.cc:492] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25: No bootstrap required, opened a new log
I20260925 14:49:54.548486 11106 tablet_bootstrap.cc:492] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: No bootstrap required, opened a new log
I20260925 14:49:54.548537 11105 ts_tablet_manager.cc:1397] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:54.548535 11106 ts_tablet_manager.cc:1397] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:54.550446 11109 tablet_bootstrap.cc:492] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Bootstrap starting.
I20260925 14:49:54.550495 11106 raft_consensus.cc:348] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.550495 11105 raft_consensus.cc:348] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.550685 11106 raft_consensus.cc:374] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:54.550685 11105 raft_consensus.cc:374] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:54.550722 11106 raft_consensus.cc:729] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45acaadd398f4136ab44718e997301a3, State: Initialized, Role: FOLLOWER
I20260925 14:49:54.550722 11105 raft_consensus.cc:729] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e95d1ea115da455999be13b6b12a1d25, State: Initialized, Role: FOLLOWER
I20260925 14:49:54.550840 11105 consensus_queue.cc:260] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.550854 11106 consensus_queue.cc:260] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.551121 11105 ts_tablet_manager.cc:1428] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:54.551121 11106 ts_tablet_manager.cc:1428] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:49:54.551138 11109 tablet_bootstrap.cc:654] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Neither blocks nor log segments found. Creating new log.
I20260925 14:49:54.551137 10823 heartbeater.cc:503] Master 127.0.98.62:39381 was elected leader, sending a full tablet report...
I20260925 14:49:54.551236 11091 heartbeater.cc:503] Master 127.0.98.62:39381 was elected leader, sending a full tablet report...
I20260925 14:49:54.551682 11109 log.cc:824] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:49:54.552515 11109 tablet_bootstrap.cc:492] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: No bootstrap required, opened a new log
I20260925 14:49:54.552582 11109 ts_tablet_manager.cc:1397] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:49:54.553881 11109 raft_consensus.cc:348] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.554059 11109 raft_consensus.cc:374] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:49:54.554105 11109 raft_consensus.cc:729] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05dd1448cde74463a87d07b082b6b7d7, State: Initialized, Role: FOLLOWER
I20260925 14:49:54.554224 11109 consensus_queue.cc:260] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.554557 10957 heartbeater.cc:503] Master 127.0.98.62:39381 was elected leader, sending a full tablet report...
I20260925 14:49:54.554545 11109 ts_tablet_manager.cc:1428] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260925 14:49:54.604739 10958 tablet.cc:2369] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:49:54.651017 11113 raft_consensus.cc:482] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:49:54.651153 11113 raft_consensus.cc:504] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.651612 11113 leader_election.cc:288] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45acaadd398f4136ab44718e997301a3 (127.0.98.3:46003), e95d1ea115da455999be13b6b12a1d25 (127.0.98.1:37013)
I20260925 14:49:54.654982 10959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.651823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37013 (local address 127.0.98.2:54443)
0925 14:49:54.651885 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.651898 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.651942 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.652276 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.652276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.652278 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:54.652335 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.652337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.653303 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.653304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.654058 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.654060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.654733 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.654734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.654810 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.654813 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.654866 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.654898 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":3}
I20260925 14:49:54.655009 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.652000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55769 (local address 127.0.98.3:46003)
0925 14:49:54.652059 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.652061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.652153 (+ 92us) server_negotiation.cc:407] Connection header received
0925 14:49:54.652189 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.652189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.652202 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.652262 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:54.652499 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.653195 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.654167 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.654587 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.654678 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.654689 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.654841 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.654913 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:49:54.655009 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.652072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54443 (local address 127.0.98.1:37013)
0925 14:49:54.652154 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.652157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.652165 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:49:54.652190 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.652190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.652202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.652264 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:49:54.652401 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.653193 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.654206 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.654680 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.654735 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.654743 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.654894 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.654939 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 14:49:54.654989 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.651823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46003 (local address 127.0.98.2:55769)
0925 14:49:54.652057 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.652074 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:49:54.652143 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:49:54.652400 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:49:54.652400 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:49:54.652401 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:49:54.652437 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.652438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.653320 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.653321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.654066 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:49:54.654068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.654678 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.654679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:49:54.654755 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.654759 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:49:54.654817 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:49:54.654868 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":13,"thread_start_us":130,"threads_started":1}
I20260925 14:49:54.655448 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe95403b5fa45259fa205e139c6c99a" candidate_uuid: "05dd1448cde74463a87d07b082b6b7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95d1ea115da455999be13b6b12a1d25" is_pre_election: true
I20260925 14:49:54.655653 10777 raft_consensus.cc:2454] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05dd1448cde74463a87d07b082b6b7d7 in term 0.
I20260925 14:49:54.655618 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe95403b5fa45259fa205e139c6c99a" candidate_uuid: "05dd1448cde74463a87d07b082b6b7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45acaadd398f4136ab44718e997301a3" is_pre_election: true
I20260925 14:49:54.655745 11045 raft_consensus.cc:2454] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05dd1448cde74463a87d07b082b6b7d7 in term 0.
I20260925 14:49:54.655915 10844 leader_election.cc:302] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05dd1448cde74463a87d07b082b6b7d7, e95d1ea115da455999be13b6b12a1d25; no voters:
I20260925 14:49:54.656106 11113 raft_consensus.cc:2781] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:49:54.656157 11113 raft_consensus.cc:482] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:49:54.656172 11113 raft_consensus.cc:3037] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:54.656833 11113 raft_consensus.cc:504] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.656980 11113 leader_election.cc:288] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [CANDIDATE]: Term 1 election: Requested vote from peers 45acaadd398f4136ab44718e997301a3 (127.0.98.3:46003), e95d1ea115da455999be13b6b12a1d25 (127.0.98.1:37013)
I20260925 14:49:54.657120 11045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe95403b5fa45259fa205e139c6c99a" candidate_uuid: "05dd1448cde74463a87d07b082b6b7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45acaadd398f4136ab44718e997301a3"
I20260925 14:49:54.657150 10777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe95403b5fa45259fa205e139c6c99a" candidate_uuid: "05dd1448cde74463a87d07b082b6b7d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e95d1ea115da455999be13b6b12a1d25"
I20260925 14:49:54.657204 11045 raft_consensus.cc:3037] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:54.657254 10777 raft_consensus.cc:3037] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:49:54.657932 10777 raft_consensus.cc:2454] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05dd1448cde74463a87d07b082b6b7d7 in term 1.
I20260925 14:49:54.657980 11045 raft_consensus.cc:2454] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05dd1448cde74463a87d07b082b6b7d7 in term 1.
I20260925 14:49:54.658134 10844 leader_election.cc:302] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05dd1448cde74463a87d07b082b6b7d7, e95d1ea115da455999be13b6b12a1d25; no voters:
I20260925 14:49:54.658272 11113 raft_consensus.cc:2781] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:49:54.658490 11113 raft_consensus.cc:686] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 1 LEADER]: Becoming Leader. State: Replica: 05dd1448cde74463a87d07b082b6b7d7, State: Running, Role: LEADER
I20260925 14:49:54.658588 11113 consensus_queue.cc:237] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } }
I20260925 14:49:54.659457 10631 catalog_manager.cc:5495] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05dd1448cde74463a87d07b082b6b7d7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "05dd1448cde74463a87d07b082b6b7d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05dd1448cde74463a87d07b082b6b7d7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 } health_report { overall_health: UNKNOWN } } }
I20260925 14:49:54.690565 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.688221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.0.98.2:43887)
0925 14:49:54.688278 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:49:54.688280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:49:54.688291 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:49:54.688331 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:49:54.688332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:49:54.688351 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:49:54.688415 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:49:54.688552 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.689032 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.689764 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:49:54.689954 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:49:54.690027 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:49:54.690169 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:49:54.690271 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:49:54.690333 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:49:54.690468 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 14:49:54.690487 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:49:54.692490 11045 raft_consensus.cc:1261] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 1 FOLLOWER]: Refusing update from remote peer 05dd1448cde74463a87d07b082b6b7d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:54.692574 10777 raft_consensus.cc:1261] T 6fe95403b5fa45259fa205e139c6c99a P e95d1ea115da455999be13b6b12a1d25 [term 1 FOLLOWER]: Refusing update from remote peer 05dd1448cde74463a87d07b082b6b7d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:49:54.692862 11113 consensus_queue.cc:1035] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45acaadd398f4136ab44718e997301a3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:54.693040 11115 consensus_queue.cc:1035] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e95d1ea115da455999be13b6b12a1d25" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:49:54.695739 11119 mvcc.cc:205] Tried to move back new op lower bound from 7333264567057629184 to 7333264566922076160. Current Snapshot: MvccSnapshot[applied={T|T < 7333264567057629184}]
I20260925 14:49:54.696187 11118 mvcc.cc:205] Tried to move back new op lower bound from 7333264567057629184 to 7333264566922076160. Current Snapshot: MvccSnapshot[applied={T|T < 7333264567057629184 or (T in {7333264567057629184})}]
I20260925 14:49:54.697279 11120 mvcc.cc:205] Tried to move back new op lower bound from 7333264567057629184 to 7333264566922076160. Current Snapshot: MvccSnapshot[applied={T|T < 7333264567057629184}]
I20260925 14:49:54.697639 10631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:54.697732 10631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:49:54.698472 10631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f7270d3cfdc44a8a26af27a1bddeaa6: Sending DeleteTablet for 3 replicas of tablet 6fe95403b5fa45259fa205e139c6c99a
I20260925 14:49:54.698817 11025 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe95403b5fa45259fa205e139c6c99a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48906
I20260925 14:49:54.698820 10891 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe95403b5fa45259fa205e139c6c99a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:49:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45808
I20260925 14:49:54.699062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10694
I20260925 14:49:54.699150 11128 tablet_replica.cc:330] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: stopping tablet replica
I20260925 14:49:54.699156 11129 tablet_replica.cc:330] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: stopping tablet replica
I20260925 14:49:54.699252 11128 raft_consensus.cc:2229] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:49:54.699333 11129 raft_consensus.cc:2229] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:49:54.699347 11128 raft_consensus.cc:2258] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:54.699499 11129 raft_consensus.cc:2258] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:49:54.699927 11129 ts_tablet_manager.cc:1905] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:49:54.701498 11129 ts_tablet_manager.cc:1918] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:49:54.701581 11129 log.cc:1196] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/6fe95403b5fa45259fa205e139c6c99a
I20260925 14:49:54.701838 11129 ts_tablet_manager.cc:1939] T 6fe95403b5fa45259fa205e139c6c99a P 45acaadd398f4136ab44718e997301a3: Deleting consensus metadata
I20260925 14:49:54.703210 10618 catalog_manager.cc:4841] TS 45acaadd398f4136ab44718e997301a3 (127.0.98.3:46003): tablet 6fe95403b5fa45259fa205e139c6c99a (table test-table [id=deca427c1b1744558c49d159b24dcd56]) successfully deleted
I20260925 14:49:54.706648 11128 ts_tablet_manager.cc:1905] T 6fe95403b5fa45259fa205e139c6c99a P 05dd1448cde74463a87d07b082b6b7d7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:49:54.707046 10616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37013 (error 108)
I20260925 14:49:54.707216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10828
I20260925 14:49:54.708554 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.707554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37013 (local address 127.0.0.1:46272)
0925 14:49:54.708485 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.708525 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":910}
W20260925 14:49:54.708695 10616 catalog_manager.cc:4584] TS e95d1ea115da455999be13b6b12a1d25 (127.0.98.1:37013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fe95403b5fa45259fa205e139c6c99a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37013: connect: Connection refused (error 111)
I20260925 14:49:54.713364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10962
I20260925 14:49:54.713431 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:49:54.713318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43887 (local address 127.0.0.1:45824)
0925 14:49:54.713367 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:49:54.713401 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 14:49:54.717603 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 10600
2026-09-25T14:49:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2019 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[11133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[11133](info): set up 2 sockets
Sep 25 14:49:54 dist-test-slave-z2s5 krb5kdc[11133](info): commencing operation
krb5kdc: starting...
W20260925 14:49:55.759436 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:49:55 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347795, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:49:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:49:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 14:49:55.928992 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM77lMC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/postgres -l logfile start
2026-09-25 14:49:58.475 UTC [11163] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:49:58.475 UTC [11163] LOG: listening on IPv4 address "127.0.98.20", port 59701
2026-09-25 14:49:58.479 UTC [11163] LOG: listening on Unix socket "/tmp/.s.PGSQL.59701"
2026-09-25 14:49:58.517 UTC [11166] LOG: database system was shut down at 2026-09-25 14:49:56 UTC
2026-09-25 14:49:58.521 UTC [11163] LOG: database system is ready to accept connections
I20260925 14:49:59.459223 392 mini_postgres.cc:96] Postgres bound to 59701
2026-09-25 14:49:59.465 UTC [11174] FATAL: database "slave" does not exist
127.0.98.20:59701 - accepting connections
I20260925 14:49:59.465762 392 mini_ranger.cc:161] Starting Ranger...
I20260925 14:49:59.480139 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:49:59.975952 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:49:59.976070 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-admin
2026-09-25 14:50:00,242 [I] DB FLAVOR :POSTGRES
2026-09-25 14:50:00,242 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:50:00,242 [I] Checking connection..
2026-09-25 14:50:00,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:50:00,481 [I] Checking connection passed.
2026-09-25 14:50:00,481 [I] --------- Verifying version history table ---------
2026-09-25 14:50:00,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:50:00,713 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:50:00,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:50:00,951 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:50:00,951 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:50:00,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:50:01.196 UTC [11238] WARNING: there is no transaction in progress
2026-09-25 14:50:01,203 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:50:01,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:50:01,443 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:50:01,443 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:50:01,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:50:01,697 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:50:01,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z2s5.c.gcp-upstream.internal','N') ;"
2026-09-25 14:50:02,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:50:02,436 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:50:02,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:50:02,682 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:50:02,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:50:02,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:50:03,159 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:50:03,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:50:04.157 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.175 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.192 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.209 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.225 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.244 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.301 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.309 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.323 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.334 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.344 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.358 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.369 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.376 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.384 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:04.392 UTC [11346] WARNING: there is no transaction in progress
2026-09-25 14:50:05,091 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:50:05,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:50:05,344 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:50:05,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:50:05,598 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:50:05,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:50:05,831 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:50:05,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:50:06,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:50:06,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z2s5.c.gcp-upstream.internal';"
2026-09-25 14:50:06,565 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:50:06,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:50:06,807 [I] DB_PATCHES have already been applied
I20260925 14:50:06.812892 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM77lMC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:50:06.812958 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 14:50:06.815964 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:40381
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:50:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:50:07.970727 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.505s user 0.000s sys 0.009s
Sep 25, 2026 2:50:28 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:50:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:50:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:50:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:50:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:50:29.779155 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:50:29 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:50:29.873298 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:50:29.883355 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:50:30.761528 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:50:30.761610 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms
I20260925 14:50:30.761632 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:59701
2026-09-25 14:50:30,939 [I] DB FLAVOR :POSTGRES
2026-09-25 14:50:30,939 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:50:30,939 [I] Checking connection
2026-09-25 14:50:30,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:50:31,200 [I] connection success
2026-09-25 14:50:31,200 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:50:31,201 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:50:31,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:50:31,427 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:50:31,428 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:50:31,428 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:50:31,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:59701/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:50:31,687 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:50:32.113184 392 mini_ranger_kms.cc:310] Created kms service
I20260925 14:50:32.286571 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:50:32.329676 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:50:32 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347829, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:50:32.751866 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:50:32.751997 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:50:32.752022 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 14:50:32.755429 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11526
I20260925 14:50:32.755584 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:33381
Sep 25, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:50:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:50:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:50:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 14:50:35 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:35.019068 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.146s user 0.006s sys 0.154s
I20260925 14:50:35.019181 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:50:35.019258 392 mini_ranger_kms.cc:401] 127.0.98.20:33381/kms/v1/keys
Sep 25 14:50:35 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:50:35 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347835, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:50:38 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:50:38.073941 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33997
--ranger_config_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33381/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:38.172024 11577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:38.172233 11577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:38.172256 11577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:38.173452 11577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:50:38.173498 11577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:38.173513 11577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:38.173525 11577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:50:38.173537 11577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:50:38.175218 11577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:33381/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33997
--ranger_config_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:50:38.175462 11577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:38.175719 11577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:38.182209 11583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:38.182230 11582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:38.182380 11577 server_base.cc:1034] running on GCE node
W20260925 14:50:38.182209 11585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:38.182770 11577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:38.183105 11577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:38.184290 11577 hybrid_clock.cc:648] HybridClock initialized: now 1790347838184272 us; error 51 us; skew 500 ppm
Sep 25 14:50:38 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:38.187232 11577 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:50:38.187752 11577 webserver.cc:460] Webserver started at http://127.0.98.62:43289/ using document root <none> and password file <none>
I20260925 14:50:38.187999 11577 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:38.188040 11577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:38.188184 11577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:50:38 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:50:38.233405 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "f8d55d1b689444d4bcd49655daa86fe2"
format_stamp: "Formatted at 2026-09-25 14:50:38 on dist-test-slave-z2s5"
server_key: "591ff467bf15b453c02d1d004dc6b7ad"
server_key_iv: "3a4bbd1f063e105d2c1c051b1fa4a81e"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:38.233824 11577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "f8d55d1b689444d4bcd49655daa86fe2"
format_stamp: "Formatted at 2026-09-25 14:50:38 on dist-test-slave-z2s5"
server_key: "591ff467bf15b453c02d1d004dc6b7ad"
server_key_iv: "3a4bbd1f063e105d2c1c051b1fa4a81e"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:38.235402 11577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:50:38.249897 11592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:38.250133 11577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:50:38.250269 11577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "f8d55d1b689444d4bcd49655daa86fe2"
format_stamp: "Formatted at 2026-09-25 14:50:38 on dist-test-slave-z2s5"
server_key: "591ff467bf15b453c02d1d004dc6b7ad"
server_key_iv: "3a4bbd1f063e105d2c1c051b1fa4a81e"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:38.250407 11577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:38.284905 11577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:38.287194 11577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:38.287379 11577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:38.292088 11577 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33997
I20260925 14:50:38.292148 11644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33997 every 8 connection(s)
I20260925 14:50:38.292505 11577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:50:38.293182 11645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:38.295876 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2: Bootstrap starting.
I20260925 14:50:38.296459 11645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:38.296727 11645 log.cc:824] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:38.297456 11645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2: No bootstrap required, opened a new log
I20260925 14:50:38.299557 11645 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d55d1b689444d4bcd49655daa86fe2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 14:50:38.299700 11645 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:38.299722 11645 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8d55d1b689444d4bcd49655daa86fe2, State: Initialized, Role: FOLLOWER
I20260925 14:50:38.299877 11645 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d55d1b689444d4bcd49655daa86fe2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 14:50:38.299954 11645 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:50:38.299978 11645 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:50:38.300045 11645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:38.300580 11645 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d55d1b689444d4bcd49655daa86fe2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 14:50:38.300662 11645 leader_election.cc:302] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8d55d1b689444d4bcd49655daa86fe2; no voters:
I20260925 14:50:38.300871 11645 leader_election.cc:288] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:50:38.301097 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 11577
I20260925 14:50:38.301211 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:50:38.301406 392 external_mini_cluster.cc:1428] Setting key 7335de4d953f9e79ea07372a67ec9d87
I20260925 14:50:38.300917 11648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:38.301743 11648 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [term 1 LEADER]: Becoming Leader. State: Replica: f8d55d1b689444d4bcd49655daa86fe2, State: Running, Role: LEADER
I20260925 14:50:38.301873 11645 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:50:38.301854 11648 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d55d1b689444d4bcd49655daa86fe2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } }
I20260925 14:50:38.302438 11648 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8d55d1b689444d4bcd49655daa86fe2. Latest consensus state: current_term: 1 leader_uuid: "f8d55d1b689444d4bcd49655daa86fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d55d1b689444d4bcd49655daa86fe2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } }
I20260925 14:50:38.302575 11648 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:38.302462 11649 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8d55d1b689444d4bcd49655daa86fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8d55d1b689444d4bcd49655daa86fe2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33997 } } }
I20260925 14:50:38.302747 11649 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2 [sys.catalog]: This master's current role is: LEADER
I20260925 14:50:38.303429 11645 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:50:38.306756 11659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:50:38.307994 11659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:50:38 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:38.313859 11659 catalog_manager.cc:1347] Generated new cluster ID: 5125f38425b644998da8f3f67ced53d6
I20260925 14:50:38.313907 11659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:50:38.320098 11659 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:50:38.320856 11659 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:50:38.321398 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:38.302100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48330 (local address 127.0.98.62:33997)
0925 14:50:38.303805 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:38.303812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:38.303840 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:50:38.305583 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:38.305635 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:38.305804 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:38.306895 (+ 1091us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:38.307163 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:38.309077 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:38.310171 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:38.310812 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:38.311247 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:38.313779 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:38.313784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:38.313785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:38.313797 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:38.319324 (+ 5527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:38.319805 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:38.319808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:38.319808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:38.319932 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:38.320174 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:38.320175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:38.320175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:38.320450 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:38.320981 (+ 531us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:38.321055 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:50:38.321196 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1663,"thread_start_us":76,"threads_started":1}
I20260925 14:50:38.325129 11659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2: Generated new TSK 0
I20260925 14:50:38.325372 11659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:50:39.090617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33381/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:39.186874 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:39.187095 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:39.187131 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:39.188287 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:39.188349 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:39.188391 11690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:50:39.190256 11690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:33381/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:50:39.190553 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:39.190773 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:39.195600 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:39.195751 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:39.195600 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:39.196805 11690 server_base.cc:1034] running on GCE node
I20260925 14:50:39.197016 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:39.197285 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:39.198477 11690 hybrid_clock.cc:648] HybridClock initialized: now 1790347839198452 us; error 38 us; skew 500 ppm
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:39.201385 11690 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:50:39.201934 11690 webserver.cc:460] Webserver started at http://127.0.98.1:34581/ using document root <none> and password file <none>
I20260925 14:50:39.202198 11690 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:39.202276 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:39.202433 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:50:39.215153 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "e4d67093d85c49a58ed1c50ce2cb78fa"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "791289c080a0dbea572b380552f4176b"
server_key_iv: "77b6a2d069cc228b4fbaa4ab439ccb03"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.215567 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e4d67093d85c49a58ed1c50ce2cb78fa"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "791289c080a0dbea572b380552f4176b"
server_key_iv: "77b6a2d069cc228b4fbaa4ab439ccb03"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.217067 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:50:39.232254 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.232450 11690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:39.232555 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "e4d67093d85c49a58ed1c50ce2cb78fa"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "791289c080a0dbea572b380552f4176b"
server_key_iv: "77b6a2d069cc228b4fbaa4ab439ccb03"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.232661 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:39.248014 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:39.250456 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:39.250600 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:39.250836 11690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:39.251214 11690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:39.251258 11690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.251288 11690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:39.251307 11690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.258141 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38137
I20260925 14:50:39.258678 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:50:39.260175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 11690
I20260925 14:50:39.260273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:50:39.260454 392 external_mini_cluster.cc:1428] Setting key 5338a3eaaa8af1c07d01122f78de3d41
I20260925 14:50:39.262458 11820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38137 every 8 connection(s)
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:50:39.272352 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.261328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46769 (local address 127.0.98.62:33997)
0925 14:50:39.262696 (+ 1368us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.262699 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.262709 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:39.262749 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.262750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.262764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.263555 (+ 791us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:39.263592 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.264386 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.265880 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.266208 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.268741 (+ 2533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.268790 (+ 49us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:39.268791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:39.268791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:39.268795 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:39.270636 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.271157 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.271158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.271158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.271216 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.271556 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.271557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.271557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.271734 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:39.271801 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.272162 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.272249 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1335,"thread_start_us":71,"threads_started":1}
I20260925 14:50:39.272535 11823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.261354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.98.1:46769)
0925 14:50:39.261534 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.261588 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.262236 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.262940 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.262945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.263184 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:39.263460 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.263466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.264494 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.264495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.265781 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.265784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.266302 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.266303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.266469 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.267094 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:39.267105 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:39.268638 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:39.270743 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:39.270749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:39.270751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:39.271068 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:39.271311 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:39.271312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:39.271312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:39.271460 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:39.271796 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:39.271803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:39.271989 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.272125 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.272291 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 14:50:39.273181 11821 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33997
I20260925 14:50:39.273324 11821 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:39.273535 11821 heartbeater.cc:511] Master 127.0.98.62:33997 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 14:50:39.274112 11609 ts_manager.cc:194] Registered new tserver with Master: e4d67093d85c49a58ed1c50ce2cb78fa (127.0.98.1:38137)
I20260925 14:50:39.275084 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:50:39.292991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33381/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:39.411473 11827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:39.411751 11827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:39.411805 11827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:39.413100 11827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:39.413178 11827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:39.413229 11827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:50:39.414973 11827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:33381/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:50:39.415280 11827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:39.415560 11827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:39.421221 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:39.421574 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:39.421682 11827 server_base.cc:1034] running on GCE node
W20260925 14:50:39.421375 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:39.422132 11827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:39.422534 11827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:39.423787 11827 hybrid_clock.cc:648] HybridClock initialized: now 1790347839423765 us; error 51 us; skew 500 ppm
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:39.426522 11827 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:50:39.427023 11827 webserver.cc:460] Webserver started at http://127.0.98.2:37885/ using document root <none> and password file <none>
I20260925 14:50:39.427271 11827 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:39.427341 11827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:39.427479 11827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:50:39.439522 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "21bea38d3b084ef698ab39eed1caed6f"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "672281fd444ee4d9e25d96eaca4509d8"
server_key_iv: "6f51090d7b73c239df6f14632f099b73"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.439942 11827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21bea38d3b084ef698ab39eed1caed6f"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "672281fd444ee4d9e25d96eaca4509d8"
server_key_iv: "6f51090d7b73c239df6f14632f099b73"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.441386 11827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:39.454561 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.463833 11827 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260925 14:50:39.463989 11827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "21bea38d3b084ef698ab39eed1caed6f"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "672281fd444ee4d9e25d96eaca4509d8"
server_key_iv: "6f51090d7b73c239df6f14632f099b73"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.464115 11827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:39.508879 11827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:39.511559 11827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:39.511711 11827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:39.511924 11827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:39.512363 11827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:39.512426 11827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.512457 11827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:39.512476 11827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.519505 11827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41871
I20260925 14:50:39.519997 11827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:50:39.520653 11957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41871 every 8 connection(s)
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:39.530038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 11827
I20260925 14:50:39.530138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:50:39.530332 392 external_mini_cluster.cc:1428] Setting key 4d08abd76e64cef3c877bcc0e06f23f2
I20260925 14:50:39.532023 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.520968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52259 (local address 127.0.98.62:33997)
0925 14:50:39.521016 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.521018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.521155 (+ 137us) server_negotiation.cc:407] Connection header received
0925 14:50:39.522280 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.522280 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.522291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.522364 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:39.523111 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.523920 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.525486 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.525817 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.525899 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.528440 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:39.528442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:39.528442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:39.528451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:39.530159 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.530902 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.530904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.530904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.530964 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.531253 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.531253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.531254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.531384 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:39.531435 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.531831 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.531922 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:50:39.532754 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.520701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.98.2:52259)
0925 14:50:39.521003 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.521050 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.522153 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.522468 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.522473 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.522715 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:39.522988 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.522994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.524052 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.524054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.525381 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.525384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.525941 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.525942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.526092 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.526717 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:39.526727 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:39.528341 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:39.530517 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:39.530523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:39.530526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:39.530808 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:39.531036 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:39.531036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:39.531037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:39.531164 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:39.531476 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:39.531482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:39.531654 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.532367 (+ 713us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.532530 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":74,"threads_started":1}
I20260925 14:50:39.533542 11958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33997
I20260925 14:50:39.533703 11958 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:39.533931 11958 heartbeater.cc:511] Master 127.0.98.62:33997 requested a full tablet report, sending...
I20260925 14:50:39.534420 11609 ts_manager.cc:194] Registered new tserver with Master: 21bea38d3b084ef698ab39eed1caed6f (127.0.98.2:41871)
I20260925 14:50:39.534932 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52259
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:50:39.571143 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:33381/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:50:39.701411 11964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:50:39.701774 11964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:50:39.701880 11964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:50:39.703701 11964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:50:39.703810 11964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:50:39.703897 11964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:50:39.706118 11964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:33381/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:50:39.706633 11964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:50:39.706967 11964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:50:39.714179 11970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:39.714609 11969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:50:39.714864 11972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:50:39.715947 11964 server_base.cc:1034] running on GCE node
I20260925 14:50:39.716320 11964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:50:39.716611 11964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:50:39.717792 11964 hybrid_clock.cc:648] HybridClock initialized: now 1790347839717769 us; error 36 us; skew 500 ppm
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:50:39.721210 11964 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:50:39.721768 11964 webserver.cc:460] Webserver started at http://127.0.98.3:35589/ using document root <none> and password file <none>
I20260925 14:50:39.722038 11964 fs_manager.cc:359] Metadata directory not provided
I20260925 14:50:39.722095 11964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:50:39.722218 11964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:50:39.746426 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "171d33f7a4de43059c2017b462ea0572"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "058a7f293904d2816db4943a9cae2d54"
server_key_iv: "884d776e4c6c6b76b2cf993c477544a4"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.747015 11964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "171d33f7a4de43059c2017b462ea0572"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "058a7f293904d2816db4943a9cae2d54"
server_key_iv: "884d776e4c6c6b76b2cf993c477544a4"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.751269 11964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.001s
I20260925 14:50:39.766054 11979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.766364 11964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:50:39.766559 11964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "171d33f7a4de43059c2017b462ea0572"
format_stamp: "Formatted at 2026-09-25 14:50:39 on dist-test-slave-z2s5"
server_key: "058a7f293904d2816db4943a9cae2d54"
server_key_iv: "884d776e4c6c6b76b2cf993c477544a4"
server_key_version: "kuduclusterkey@0"
I20260925 14:50:39.766729 11964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:50:39.784917 11964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:50:39.787570 11964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:50:39.787776 11964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:50:39.788089 11964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:50:39.788498 11964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:50:39.788581 11964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.788648 11964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:50:39.788702 11964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:50:39.796344 11964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42945
I20260925 14:50:39.796831 11964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:50:39.799733 12092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42945 every 8 connection(s)
I20260925 14:50:39.804889 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 11964
I20260925 14:50:39.805346 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:50:39.805595 392 external_mini_cluster.cc:1428] Setting key 2fa05503132ef8ab479ebe10b684077e
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:50:39.810519 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.799578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58929 (local address 127.0.98.62:33997)
0925 14:50:39.799622 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.799624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.801105 (+ 1481us) server_negotiation.cc:407] Connection header received
0925 14:50:39.801687 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.801688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.801702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.801753 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:39.802426 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.803042 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.804215 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.804442 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.804529 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.807043 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:39.807045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:39.807045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:39.807053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:39.808887 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.809383 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.809385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.809385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.809452 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.809754 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.809755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.809755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.809882 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:39.809965 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.810293 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.810425 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:50:39.810710 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.799302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33997 (local address 127.0.98.3:58929)
0925 14:50:39.800938 (+ 1636us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.800989 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.801571 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.801834 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.801840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.802096 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:50:39.802294 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.802299 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.803138 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.803140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.804116 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.804120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.804541 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.804543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.804709 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.805312 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:50:39.805326 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:50:39.806950 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:50:39.809008 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:39.809016 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:39.809018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:39.809279 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:39.809527 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:50:39.809528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:50:39.809528 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:50:39.809666 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:50:39.809955 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:50:39.809962 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:50:39.810130 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.810255 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.810447 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1585,"thread_start_us":84,"threads_started":1}
I20260925 14:50:39.811347 12093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33997
I20260925 14:50:39.811474 12093 heartbeater.cc:464] Registering TS with master...
I20260925 14:50:39.811671 12093 heartbeater.cc:511] Master 127.0.98.62:33997 requested a full tablet report, sending...
I20260925 14:50:39.812088 11609 ts_manager.cc:194] Registered new tserver with Master: 171d33f7a4de43059c2017b462ea0572 (127.0.98.3:42945)
I20260925 14:50:39.812487 11609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58929
I20260925 14:50:39.817198 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:50:39.824988 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.818653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.0.98.62:33997)
0925 14:50:39.818699 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.818701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.818837 (+ 136us) server_negotiation.cc:407] Connection header received
0925 14:50:39.818910 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.818911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.818919 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.818972 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:50:39.819525 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.820008 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.820776 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.821037 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.821094 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.822356 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:50:39.822357 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:50:39.822358 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:50:39.822365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:50:39.823834 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.824202 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.824203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.824203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.824257 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:50:39.824471 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:50:39.824472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:50:39.824472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:50:39.824587 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:50:39.824695 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.824794 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.824879 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:50:39.826865 11609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:50:39.828642 11609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:50:39.839028 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.831071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38137 (local address 127.0.0.1:33426)
0925 14:50:39.831297 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.831356 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.831422 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.835035 (+ 3613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.835036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.835038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:39.835080 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.835082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.836578 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.836579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.837702 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.837705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.838740 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.838741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.838829 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.838846 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.838897 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.838945 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260925 14:50:39.839738 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.831166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.98.1:38137)
0925 14:50:39.834850 (+ 3684us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.834856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.834868 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:50:39.834902 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.834904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.834919 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.835150 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:39.835214 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.836480 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.837810 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.838628 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.839523 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.839602 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.839629 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.839669 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3645,"thread_start_us":72,"threads_started":1}
I20260925 14:50:39.840555 11755 tablet_service.cc:1467] Processing CreateTablet for tablet 8feff2719381477cbf08991add76ea33 (DEFAULT_TABLE table=test-table [id=64aa1dacb25a4e3198e80604a4dea0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:39.840901 11755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff2719381477cbf08991add76ea33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:39.852039 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.831308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42945 (local address 127.0.0.1:55320)
0925 14:50:39.832135 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.832146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.832194 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.832454 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.832455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.832457 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:39.832503 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.832504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.833956 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.833957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.850511 (+ 16554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.850518 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.851764 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.851766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.851840 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.851845 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.851895 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.851946 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"thread_start_us":128,"threads_started":1}
I20260925 14:50:39.852514 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.831452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.98.3:42945)
0925 14:50:39.832081 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.832085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.832246 (+ 161us) server_negotiation.cc:407] Connection header received
0925 14:50:39.832284 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.832286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.832301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.832380 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:39.832574 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.833593 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.850673 (+ 17080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.851557 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.851641 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.851720 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.852385 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.852431 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":82,"threads_started":1}
I20260925 14:50:39.853470 12027 tablet_service.cc:1467] Processing CreateTablet for tablet 8feff2719381477cbf08991add76ea33 (DEFAULT_TABLE table=test-table [id=64aa1dacb25a4e3198e80604a4dea0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:39.853829 12027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff2719381477cbf08991add76ea33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:39.855604 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.831071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41871 (local address 127.0.0.1:36030)
0925 14:50:39.831532 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.831541 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.831577 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.831916 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.831919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.831922 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:39.832000 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.832003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.833382 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.833382 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.834728 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.834730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.855341 (+ 20611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.855343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.855422 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.855429 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.855463 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.855513 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260925 14:50:39.855963 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.831579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36030 (local address 127.0.98.2:41871)
0925 14:50:39.831721 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.831727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.831737 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:39.831777 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.831779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.831795 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.832089 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:39.832164 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.833286 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.854333 (+ 21047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.855189 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.855707 (+ 518us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.855803 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.855833 (+ 30us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.855886 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 14:50:39.856354 12111 tablet_bootstrap.cc:492] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa: Bootstrap starting.
I20260925 14:50:39.857123 12111 tablet_bootstrap.cc:654] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:39.857434 12111 log.cc:824] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:39.858307 11874 tablet_service.cc:1467] Processing CreateTablet for tablet 8feff2719381477cbf08991add76ea33 (DEFAULT_TABLE table=test-table [id=64aa1dacb25a4e3198e80604a4dea0b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:50:39.858670 11874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8feff2719381477cbf08991add76ea33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:50:39.862847 12113 tablet_bootstrap.cc:492] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: Bootstrap starting.
I20260925 14:50:39.863943 12113 tablet_bootstrap.cc:654] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:39.864164 12112 tablet_bootstrap.cc:492] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Bootstrap starting.
I20260925 14:50:39.865154 12112 tablet_bootstrap.cc:654] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Neither blocks nor log segments found. Creating new log.
I20260925 14:50:39.865585 12113 log.cc:824] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:39.866011 12112 log.cc:824] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Log is configured to *not* fsync() on all Append() calls
I20260925 14:50:39.867019 12113 tablet_bootstrap.cc:492] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: No bootstrap required, opened a new log
I20260925 14:50:39.867131 12113 ts_tablet_manager.cc:1397] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:50:39.868604 12111 tablet_bootstrap.cc:492] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa: No bootstrap required, opened a new log
I20260925 14:50:39.868784 12111 ts_tablet_manager.cc:1397] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20260925 14:50:39.869666 12113 raft_consensus.cc:348] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.869932 12113 raft_consensus.cc:374] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:39.870009 12113 raft_consensus.cc:729] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21bea38d3b084ef698ab39eed1caed6f, State: Initialized, Role: FOLLOWER
I20260925 14:50:39.870196 12113 consensus_queue.cc:260] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.870596 12113 ts_tablet_manager.cc:1428] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:39.870779 11958 heartbeater.cc:503] Master 127.0.98.62:33997 was elected leader, sending a full tablet report...
I20260925 14:50:39.871368 12112 tablet_bootstrap.cc:492] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: No bootstrap required, opened a new log
I20260925 14:50:39.871426 12112 ts_tablet_manager.cc:1397] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 14:50:39.873186 12111 raft_consensus.cc:348] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.873421 12111 raft_consensus.cc:374] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:39.873502 12111 raft_consensus.cc:729] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4d67093d85c49a58ed1c50ce2cb78fa, State: Initialized, Role: FOLLOWER
I20260925 14:50:39.873688 12111 consensus_queue.cc:260] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.873780 12112 raft_consensus.cc:348] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.873996 12112 raft_consensus.cc:374] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:50:39.874027 12111 ts_tablet_manager.cc:1428] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260925 14:50:39.874068 12112 raft_consensus.cc:729] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 171d33f7a4de43059c2017b462ea0572, State: Initialized, Role: FOLLOWER
I20260925 14:50:39.874200 11821 heartbeater.cc:503] Master 127.0.98.62:33997 was elected leader, sending a full tablet report...
I20260925 14:50:39.874246 12112 consensus_queue.cc:260] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.874501 12112 ts_tablet_manager.cc:1428] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:50:39.874598 12093 heartbeater.cc:503] Master 127.0.98.62:33997 was elected leader, sending a full tablet report...
I20260925 14:50:39.947557 12117 raft_consensus.cc:482] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:50:39.947698 12117 raft_consensus.cc:504] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.948115 12117 leader_election.cc:288] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4d67093d85c49a58ed1c50ce2cb78fa (127.0.98.1:38137), 171d33f7a4de43059c2017b462ea0572 (127.0.98.3:42945)
I20260925 14:50:39.952241 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.948411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38137 (local address 127.0.98.2:58935)
0925 14:50:39.948510 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.948526 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.948594 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.948849 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.948850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.948852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:39.948935 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.948937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.949878 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.949878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.951130 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.951133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.951982 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.951983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.952062 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.952066 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.952119 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.952163 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 14:50:39.952749 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.948643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58935 (local address 127.0.98.1:38137)
0925 14:50:39.948686 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.948688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.948698 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:50:39.948728 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.948728 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.948737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.948797 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:39.949019 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.949752 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.951251 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.951853 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.951936 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.951949 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.952635 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.952682 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:50:39.953096 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff2719381477cbf08991add76ea33" candidate_uuid: "21bea38d3b084ef698ab39eed1caed6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" is_pre_election: true
I20260925 14:50:39.953251 11768 raft_consensus.cc:2454] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bea38d3b084ef698ab39eed1caed6f in term 0.
I20260925 14:50:39.953866 11847 leader_election.cc:302] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21bea38d3b084ef698ab39eed1caed6f, e4d67093d85c49a58ed1c50ce2cb78fa; no voters:
I20260925 14:50:39.953934 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.949935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60749 (local address 127.0.98.3:42945)
0925 14:50:39.949974 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.949976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.950107 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:50:39.950198 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.950199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.950209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.950237 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:50:39.950465 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.951194 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.953003 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.953540 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.953582 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.953591 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.953815 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.953867 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:50:39.954172 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.949747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42945 (local address 127.0.98.2:60749)
0925 14:50:39.950050 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:39.950060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:50:39.950129 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:50:39.950297 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:50:39.950298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:50:39.950299 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:50:39.950369 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.950370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.951304 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.951305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.952900 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:50:39.952903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.953660 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.953661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:50:39.953737 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.953740 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:50:39.954044 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 14:50:39.954085 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":86,"threads_started":1}
I20260925 14:50:39.954170 12117 raft_consensus.cc:2781] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:50:39.954365 12117 raft_consensus.cc:482] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:50:39.954481 12117 raft_consensus.cc:3037] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:39.954572 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff2719381477cbf08991add76ea33" candidate_uuid: "21bea38d3b084ef698ab39eed1caed6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171d33f7a4de43059c2017b462ea0572" is_pre_election: true
I20260925 14:50:39.954705 12046 raft_consensus.cc:2454] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21bea38d3b084ef698ab39eed1caed6f in term 0.
I20260925 14:50:39.955422 12117 raft_consensus.cc:504] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.955555 12117 leader_election.cc:288] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [CANDIDATE]: Term 1 election: Requested vote from peers e4d67093d85c49a58ed1c50ce2cb78fa (127.0.98.1:38137), 171d33f7a4de43059c2017b462ea0572 (127.0.98.3:42945)
I20260925 14:50:39.955752 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff2719381477cbf08991add76ea33" candidate_uuid: "21bea38d3b084ef698ab39eed1caed6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa"
I20260925 14:50:39.955788 12046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8feff2719381477cbf08991add76ea33" candidate_uuid: "21bea38d3b084ef698ab39eed1caed6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "171d33f7a4de43059c2017b462ea0572"
I20260925 14:50:39.955821 11768 raft_consensus.cc:3037] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:39.955860 12046 raft_consensus.cc:3037] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:50:39.956672 12046 raft_consensus.cc:2454] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bea38d3b084ef698ab39eed1caed6f in term 1.
I20260925 14:50:39.956682 11768 raft_consensus.cc:2454] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21bea38d3b084ef698ab39eed1caed6f in term 1.
I20260925 14:50:39.956869 11848 leader_election.cc:302] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 171d33f7a4de43059c2017b462ea0572, 21bea38d3b084ef698ab39eed1caed6f; no voters:
I20260925 14:50:39.957108 12117 raft_consensus.cc:2781] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:50:39.957237 12117 raft_consensus.cc:686] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [term 1 LEADER]: Becoming Leader. State: Replica: 21bea38d3b084ef698ab39eed1caed6f, State: Running, Role: LEADER
I20260925 14:50:39.957319 12117 consensus_queue.cc:237] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } }
I20260925 14:50:39.958150 11609 catalog_manager.cc:5495] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 21bea38d3b084ef698ab39eed1caed6f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "21bea38d3b084ef698ab39eed1caed6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21bea38d3b084ef698ab39eed1caed6f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 } health_report { overall_health: UNKNOWN } } }
I20260925 14:50:39.990770 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:39.988184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.0.98.2:41871)
0925 14:50:39.988317 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 14:50:39.988319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:50:39.988325 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:50:39.988357 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:50:39.988358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:50:39.988379 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:50:39.988457 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:50:39.988613 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.989187 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.989956 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:50:39.990163 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:50:39.990249 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:50:39.990428 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:50:39.990533 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:50:39.990631 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 14:50:39.990656 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 14:50:39.990676 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 14:50:39.992561 12046 raft_consensus.cc:1261] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 1 FOLLOWER]: Refusing update from remote peer 21bea38d3b084ef698ab39eed1caed6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:39.992591 11768 raft_consensus.cc:1261] T 8feff2719381477cbf08991add76ea33 P e4d67093d85c49a58ed1c50ce2cb78fa [term 1 FOLLOWER]: Refusing update from remote peer 21bea38d3b084ef698ab39eed1caed6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:50:39.993026 12117 consensus_queue.cc:1035] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4d67093d85c49a58ed1c50ce2cb78fa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:39.993196 12117 consensus_queue.cc:1035] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "171d33f7a4de43059c2017b462ea0572" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:50:39.998140 12126 mvcc.cc:205] Tried to move back new op lower bound from 7333264752607072256 to 7333264752465952768. Current Snapshot: MvccSnapshot[applied={T|T < 7333264752607072256}]
Sep 25 14:50:39 dist-test-slave-z2s5 krb5kdc[11133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:50:40.000972 11608 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:40.001085 11608 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:50:40.001791 11608 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8d55d1b689444d4bcd49655daa86fe2: Sending DeleteTablet for 3 replicas of tablet 8feff2719381477cbf08991add76ea33
I20260925 14:50:40.002151 11755 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feff2719381477cbf08991add76ea33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33426
I20260925 14:50:40.002285 12027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feff2719381477cbf08991add76ea33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55320
I20260925 14:50:40.002532 11874 tablet_service.cc:1514] Processing DeleteTablet for tablet 8feff2719381477cbf08991add76ea33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:50:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36030
I20260925 14:50:40.002627 12134 tablet_replica.cc:330] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: stopping tablet replica
I20260925 14:50:40.002724 12134 raft_consensus.cc:2229] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:50:40.003064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 11690
W20260925 14:50:40.022255 11595 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38137 (error 108)
I20260925 14:50:40.022456 12134 raft_consensus.cc:2258] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:50:40.022713 12135 tablet_replica.cc:330] T 8feff2719381477cbf08991add76ea33 P 21bea38d3b084ef698ab39eed1caed6f: stopping tablet replica
I20260925 14:50:40.022843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 11827
I20260925 14:50:40.023818 12134 ts_tablet_manager.cc:1905] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:50:40.025498 12134 ts_tablet_manager.cc:1918] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:50:40.025573 12134 log.cc:1196] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/8feff2719381477cbf08991add76ea33
I20260925 14:50:40.025843 12134 ts_tablet_manager.cc:1939] T 8feff2719381477cbf08991add76ea33 P 171d33f7a4de43059c2017b462ea0572: Deleting consensus metadata
I20260925 14:50:40.027073 11596 catalog_manager.cc:4841] TS 171d33f7a4de43059c2017b462ea0572 (127.0.98.3:42945): tablet 8feff2719381477cbf08991add76ea33 (table test-table [id=64aa1dacb25a4e3198e80604a4dea0b2]) successfully deleted
I20260925 14:50:40.027632 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:40.027505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38137 (local address 127.0.0.1:33436)
0925 14:50:40.027558 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:40.027598 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:50:40.027873 11595 catalog_manager.cc:4584] TS e4d67093d85c49a58ed1c50ce2cb78fa (127.0.98.1:38137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8feff2719381477cbf08991add76ea33: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38137: connect: Connection refused (error 111)
I20260925 14:50:40.029693 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:50:40.029594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41871 (local address 127.0.0.1:36048)
0925 14:50:40.029635 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:50:40.029657 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 14:50:40.033560 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 11964
I20260925 14:50:40.039697 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 11577
I20260925 14:50:40.044842 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:50:40.393519 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:50:40.393638 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:50:40.738759 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:50:40.739 UTC [11163] LOG: received smart shutdown request
2026-09-25 14:50:40.742 UTC [11163] LOG: background worker "logical replication launcher" (PID 11172) exited with exit code 1
2026-09-25 14:50:40.743 UTC [11167] LOG: shutting down
2026-09-25 14:50:40.930 UTC [11163] LOG: database system is shut down
2026-09-25T14:50:40Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46329 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:50:41 dist-test-slave-z2s5 krb5kdc[12145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:50:41 dist-test-slave-z2s5 krb5kdc[12145](info): set up 2 sockets
Sep 25 14:50:41 dist-test-slave-z2s5 krb5kdc[12145](info): commencing operation
krb5kdc: starting...
W20260925 14:50:42.118307 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:50:42 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:50:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260925 14:50:42.285327 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskM77lMC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/postgres -l logfile start
2026-09-25 14:50:44.826 UTC [12175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:50:44.826 UTC [12175] LOG: listening on IPv4 address "127.0.98.20", port 37675
2026-09-25 14:50:44.830 UTC [12175] LOG: listening on Unix socket "/tmp/.s.PGSQL.37675"
2026-09-25 14:50:44.851 UTC [12178] LOG: database system was shut down at 2026-09-25 14:50:43 UTC
2026-09-25 14:50:44.854 UTC [12175] LOG: database system is ready to accept connections
I20260925 14:50:45.826670 392 mini_postgres.cc:96] Postgres bound to 37675
2026-09-25 14:50:45.833 UTC [12186] FATAL: database "slave" does not exist
127.0.98.20:37675 - accepting connections
I20260925 14:50:45.833709 392 mini_ranger.cc:161] Starting Ranger...
I20260925 14:50:45.848344 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:50:46.356541 392 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:50:46.356634 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-admin
2026-09-25 14:50:46,633 [I] DB FLAVOR :POSTGRES
2026-09-25 14:50:46,634 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:50:46,634 [I] Checking connection..
2026-09-25 14:50:46,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:50:46,878 [I] Checking connection passed.
2026-09-25 14:50:46,878 [I] --------- Verifying version history table ---------
2026-09-25 14:50:46,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:50:47,151 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:50:47,151 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:50:47,398 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:50:47,398 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:50:47,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:50:47.674 UTC [12250] WARNING: there is no transaction in progress
2026-09-25 14:50:47,681 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:50:47,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:50:47,941 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:50:47,942 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:50:47,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:50:48,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:50:48,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-z2s5.c.gcp-upstream.internal','N') ;"
2026-09-25 14:50:48,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:50:48,938 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:50:48,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:50:49,179 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:50:49,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:50:49,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:50:49,650 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:50:49,651 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:50:50.632 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.649 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.666 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.683 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.699 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.718 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.777 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.784 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.799 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.809 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.820 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.836 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.847 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.853 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.861 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:50.868 UTC [12358] WARNING: there is no transaction in progress
2026-09-25 14:50:51,567 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:50:51,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:50:51,824 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:50:51,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:50:52,088 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:50:52,089 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:50:52,343 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:50:52,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:50:52,588 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:50:52,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-z2s5.c.gcp-upstream.internal';"
2026-09-25 14:50:53,092 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:50:53,093 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:50:53,351 [I] DB_PATCHES have already been applied
I20260925 14:50:53.356451 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskM77lMC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:50:53.356539 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260925 14:50:53.359573 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55365
Sep 25, 2026 2:50:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:50:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:50:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:50:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:50:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:50:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:50:54.489496 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.656s user 0.000s sys 0.009s
Sep 25, 2026 2:51:15 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:51:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:51:15 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:51:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:51:16 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:51:16.972944 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:51:17 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:51:17.059103 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:51:17.068581 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:51:17.967221 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:51:17.967305 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms
I20260925 14:51:17.967327 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:37675
2026-09-25 14:51:18,126 [I] DB FLAVOR :POSTGRES
2026-09-25 14:51:18,126 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:51:18,126 [I] Checking connection
2026-09-25 14:51:18,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:51:18,370 [I] connection success
2026-09-25 14:51:18,371 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:51:18,371 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:51:18,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:51:18,608 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:51:18,608 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:51:18,608 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:51:18,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:37675/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:51:18,886 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:51:19.353152 392 mini_ranger_kms.cc:310] Created kms service
I20260925 14:51:19.551493 392 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:51:19.595618 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:51:19 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347877, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:19.999275 392 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:51:19.999394 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskM77lMC/build/release/bin/postgresql.jar:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskM77lMC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:51:19.999415 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260925 14:51:20.002596 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12538
I20260925 14:51:20.002684 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46307
Sep 25, 2026 2:51:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:51:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:51:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:51:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:51:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:51:21 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 14:51:22.284335 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.225s user 0.005s sys 0.137s
I20260925 14:51:22.284438 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:51:22.284497 392 mini_ranger_kms.cc:401] 127.0.98.20:46307/kms/v1/keys
Sep 25 14:51:22 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347882, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 14:51:22 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347882, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:51:23 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347882, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 25 14:51:25 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:25.477837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34745
--ranger_config_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:25.584749 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:25.584965 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:25.584986 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:25.586109 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:25.586151 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:25.586165 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:25.586177 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:25.586189 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:25.587795 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34745
--ranger_config_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:25.588040 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:25.588254 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:25.593552 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:25.593561 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:25.593577 12589 server_base.cc:1034] running on GCE node
W20260925 14:51:25.593672 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:25.594099 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:25.594439 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:25.595592 12589 hybrid_clock.cc:648] HybridClock initialized: now 1790347885595581 us; error 28 us; skew 500 ppm
Sep 25 14:51:25 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:25.605152 12589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:25.605679 12589 webserver.cc:460] Webserver started at http://127.0.98.62:45249/ using document root <none> and password file <none>
I20260925 14:51:25.605938 12589 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:25.605979 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:25.606113 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:51:25 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:25.651728 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "0911cbc6d3454fd59e810f49571a1b2f"
format_stamp: "Formatted at 2026-09-25 14:51:25 on dist-test-slave-z2s5"
server_key: "dd22169a0c09df2f1a68af8764023f4f"
server_key_iv: "84ce6834681b61e774601015244e2658"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:25.652184 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "0911cbc6d3454fd59e810f49571a1b2f"
format_stamp: "Formatted at 2026-09-25 14:51:25 on dist-test-slave-z2s5"
server_key: "dd22169a0c09df2f1a68af8764023f4f"
server_key_iv: "84ce6834681b61e774601015244e2658"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:25.654086 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 14:51:25.668680 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:25.668915 12589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:25.669035 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "0911cbc6d3454fd59e810f49571a1b2f"
format_stamp: "Formatted at 2026-09-25 14:51:25 on dist-test-slave-z2s5"
server_key: "dd22169a0c09df2f1a68af8764023f4f"
server_key_iv: "84ce6834681b61e774601015244e2658"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:25.669140 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:25.692174 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:25.694553 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:25.694705 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:25.699698 12589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34745
I20260925 14:51:25.699931 12656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34745 every 8 connection(s)
I20260925 14:51:25.700245 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:25.700997 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:25.703353 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Bootstrap starting.
I20260925 14:51:25.703917 12657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:25.704200 12657 log.cc:824] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:25.704803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 12589
I20260925 14:51:25.704908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:25.704943 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: No bootstrap required, opened a new log
I20260925 14:51:25.705114 392 external_mini_cluster.cc:1428] Setting key f7083cb02623f505304285ad4e281565
I20260925 14:51:25.707430 12657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:25.707679 12657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:25.707738 12657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0911cbc6d3454fd59e810f49571a1b2f, State: Initialized, Role: FOLLOWER
I20260925 14:51:25.707916 12657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:25.708015 12657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:25.708068 12657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:25.708139 12657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:25.708849 12657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:25.708957 12657 leader_election.cc:302] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0911cbc6d3454fd59e810f49571a1b2f; no voters:
I20260925 14:51:25.709161 12657 leader_election.cc:288] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:25.709221 12662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:25.709460 12662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 LEADER]: Becoming Leader. State: Replica: 0911cbc6d3454fd59e810f49571a1b2f, State: Running, Role: LEADER
I20260925 14:51:25.709604 12662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:25.709479 12657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:25.710032 12664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0911cbc6d3454fd59e810f49571a1b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260925 14:51:25.710110 12664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:25.710356 12663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0911cbc6d3454fd59e810f49571a1b2f. Latest consensus state: current_term: 1 leader_uuid: "0911cbc6d3454fd59e810f49571a1b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260925 14:51:25.710438 12663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:25.711192 12666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:25.711568 12657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:51:25.711701 12666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:51:25 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:25.718464 12666 catalog_manager.cc:1347] Generated new cluster ID: cfcc5e7a1f574d389f10d3ebdda0a89f
I20260925 14:51:25.718532 12666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:25.718627 12661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:25.705650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.62:34745)
0925 14:51:25.705836 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:25.705846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:25.706615 (+ 769us) server_negotiation.cc:407] Connection header received
0925 14:51:25.707400 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:25.707410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:25.707473 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:25.707622 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:25.708398 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:25.709767 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:25.710925 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:25.711547 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:25.711649 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:25.714737 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:25.714741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:25.714741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:25.714754 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:25.716755 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:25.717184 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:25.717187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:25.717188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:25.717303 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:25.717553 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:25.717554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:25.717555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:25.717944 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:25.718108 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:25.718232 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 14:51:25.718355 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":79,"threads_started":1}
I20260925 14:51:25.732376 12666 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:25.733258 12666 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:25.743444 12666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Generated new TSK 0
I20260925 14:51:25.743747 12666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:51:26.493388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34745
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:26.589840 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:26.590040 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:26.590063 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:26.591287 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:26.591333 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:26.591359 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:26.592823 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:26.593052 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:26.593309 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:26.599416 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:26.599426 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:26.599499 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:26.599821 12702 server_base.cc:1034] running on GCE node
I20260925 14:51:26.600092 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:26.600374 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:26.601569 12702 hybrid_clock.cc:648] HybridClock initialized: now 1790347886601553 us; error 28 us; skew 500 ppm
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:26.604848 12702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:26.605383 12702 webserver.cc:460] Webserver started at http://127.0.98.1:37737/ using document root <none> and password file <none>
I20260925 14:51:26.605638 12702 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:26.605696 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:26.605873 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:26.620417 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "97c8be7f4f3844ada73b3edd0a0635ca"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "14972f6a904951ac7d4b8fe25d4521ea"
server_key_iv: "b462ccc9b4798f54e8eae2ef8d894609"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:26.620919 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "97c8be7f4f3844ada73b3edd0a0635ca"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "14972f6a904951ac7d4b8fe25d4521ea"
server_key_iv: "b462ccc9b4798f54e8eae2ef8d894609"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:26.622757 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:26.636806 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:26.637010 12702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:26.637109 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "97c8be7f4f3844ada73b3edd0a0635ca"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "14972f6a904951ac7d4b8fe25d4521ea"
server_key_iv: "b462ccc9b4798f54e8eae2ef8d894609"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:26.637207 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:26.665549 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:26.667798 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:26.667972 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:26.668231 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:26.668548 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:26.668603 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:26.668645 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:26.668672 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:26.675452 12702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43641
I20260925 14:51:26.675745 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43641 every 8 connection(s)
I20260925 14:51:26.675908 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:26.680855 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 12702
I20260925 14:51:26.680961 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:26.681180 392 external_mini_cluster.cc:1428] Setting key 3ebd0540ba637b865761a5c8776f0bc0
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:26.689936 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:26.676922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.98.1:55117)
0925 14:51:26.677160 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:26.677211 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:26.677848 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:26.678152 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:26.678157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:26.678423 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:26.678695 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:26.678702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.679752 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.679753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:26.681063 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:26.681065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.681701 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.681702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:26.681961 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:26.682657 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:26.682668 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:26.684333 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:26.687988 (+ 3655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:26.687998 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:26.688001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:26.688358 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:26.688623 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:26.688624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:26.688624 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:26.688804 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:26.689138 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:26.689146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:26.689351 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 14:51:26.689484 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 14:51:26.689664 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:51:26.690487 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:26.676945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55117 (local address 127.0.98.62:34745)
0925 14:51:26.677089 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:26.677091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:26.677322 (+ 231us) server_negotiation.cc:407] Connection header received
0925 14:51:26.677973 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:26.677974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:26.677992 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:26.678081 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:26.678810 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.679615 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.681164 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.681550 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.681613 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:26.684449 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:26.684451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:26.684452 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:26.684460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:26.687872 (+ 3412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:26.688457 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:26.688459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:26.688459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:26.688535 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:26.688888 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:26.688890 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:26.688890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:26.689052 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:26.689127 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:26.690259 (+ 1132us) server_negotiation.cc:299] Negotiation successful
0925 14:51:26.690355 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260925 14:51:26.690912 12833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34745
I20260925 14:51:26.691020 12833 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:26.691237 12833 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:26.691908 12621 ts_manager.cc:194] Registered new tserver with Master: 97c8be7f4f3844ada73b3edd0a0635ca (127.0.98.1:43641)
I20260925 14:51:26.692853 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:51:26.713459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34745
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:26.833791 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:26.834064 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:26.834100 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:51:26.835846 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:26.835908 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:26.835950 12839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:26.838287 12839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:26.838557 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:26.838879 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:26.845338 12849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:26.846278 12839 server_base.cc:1034] running on GCE node
W20260925 14:51:26.846583 12846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:26.846956 12847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:26.847234 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:26.847507 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:26.848690 12839 hybrid_clock.cc:648] HybridClock initialized: now 1790347886848667 us; error 41 us; skew 500 ppm
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:26.852216 12839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:26.852830 12839 webserver.cc:460] Webserver started at http://127.0.98.2:46025/ using document root <none> and password file <none>
I20260925 14:51:26.853081 12839 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:26.853133 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:26.853266 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:26.870850 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "98cef263518c4f399880f648060c8782"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "d0c24b256b8a6ddff050bdef37c46b77"
server_key_iv: "0ba6a0a96b18073d4cea40157bb8c2f8"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:26.871304 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "98cef263518c4f399880f648060c8782"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "d0c24b256b8a6ddff050bdef37c46b77"
server_key_iv: "0ba6a0a96b18073d4cea40157bb8c2f8"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:26.873661 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:26.893498 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:26.893759 12839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260925 14:51:26.893852 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "98cef263518c4f399880f648060c8782"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "d0c24b256b8a6ddff050bdef37c46b77"
server_key_iv: "0ba6a0a96b18073d4cea40157bb8c2f8"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:26.893946 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:26.923185 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:26.925598 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:26.925789 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:26.926049 12839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:26.926479 12839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:26.926532 12839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:26.926563 12839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:26.926612 12839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:26.933696 12839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38257
I20260925 14:51:26.934136 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:26.934562 12970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38257 every 8 connection(s)
Sep 25 14:51:26 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:26.943287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 12839
I20260925 14:51:26.943380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:26.943579 392 external_mini_cluster.cc:1428] Setting key fae8610f41a047f5da7a97c51dee415d
I20260925 14:51:26.947232 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:26.935227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46529 (local address 127.0.98.62:34745)
0925 14:51:26.935272 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:26.935274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:26.935286 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:26.935930 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:26.935932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:26.935945 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:26.936025 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:26.936781 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.937598 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.939120 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.939440 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.939530 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:26.942071 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:26.942073 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:26.942073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:26.942082 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:26.945232 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:26.945889 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:26.945891 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:26.945892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:26.945978 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:26.946326 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:26.946328 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:26.946328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:26.946519 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:26.946569 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:26.947024 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 14:51:26.947125 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:51:26.947444 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:26.934928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.98.2:46529)
0925 14:51:26.935119 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:26.935167 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:26.935797 (+ 630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:26.936105 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:26.936109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:26.936381 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:26.936661 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:26.936668 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.937717 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.937718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:26.939016 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:26.939019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:26.939563 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:26.939564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:26.939711 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:26.940380 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:26.940391 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:26.941979 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:26.945341 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:26.945351 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:26.945354 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:26.945733 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:26.946071 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:26.946071 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:26.946072 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:26.946247 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:26.946623 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:26.946630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:26.946851 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 14:51:26.946978 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 14:51:26.947157 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":91,"threads_started":1}
I20260925 14:51:26.948127 12971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34745
I20260925 14:51:26.948252 12971 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:26.948496 12971 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:26.948990 12621 ts_manager.cc:194] Registered new tserver with Master: 98cef263518c4f399880f648060c8782 (127.0.98.2:38257)
I20260925 14:51:26.949561 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46529
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:51:26.973419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34745
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:27.091035 12976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:27.091297 12976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:27.091341 12976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:27.092564 12976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:27.092635 12976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:27.092680 12976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:27.094183 12976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:27.094489 12976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:27.094748 12976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:27.099829 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.099835 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:27.100076 12976 server_base.cc:1034] running on GCE node
W20260925 14:51:27.099843 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:27.100414 12976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:27.100664 12976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:27.101830 12976 hybrid_clock.cc:648] HybridClock initialized: now 1790347887101820 us; error 32 us; skew 500 ppm
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:27.104449 12976 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:27.104982 12976 webserver.cc:460] Webserver started at http://127.0.98.3:37615/ using document root <none> and password file <none>
I20260925 14:51:27.105225 12976 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:27.105296 12976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:27.105443 12976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:27.118688 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "1525a7c599f245d3bfeed6527cabed42"
format_stamp: "Formatted at 2026-09-25 14:51:27 on dist-test-slave-z2s5"
server_key: "fc3737fc35ce1125ee36994b356b91c2"
server_key_iv: "11c5218af8345a85c64174066d77ee60"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.119473 12976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1525a7c599f245d3bfeed6527cabed42"
format_stamp: "Formatted at 2026-09-25 14:51:27 on dist-test-slave-z2s5"
server_key: "fc3737fc35ce1125ee36994b356b91c2"
server_key_iv: "11c5218af8345a85c64174066d77ee60"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.139062 12976 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.001s sys 0.000s
I20260925 14:51:27.158418 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.159001 12976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:27.159205 12976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "1525a7c599f245d3bfeed6527cabed42"
format_stamp: "Formatted at 2026-09-25 14:51:27 on dist-test-slave-z2s5"
server_key: "fc3737fc35ce1125ee36994b356b91c2"
server_key_iv: "11c5218af8345a85c64174066d77ee60"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.159382 12976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:27.176424 12976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:27.179081 12976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:27.179311 12976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:27.179569 12976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:27.180002 12976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:27.180100 12976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.180174 12976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:27.180234 12976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.187526 12976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34749
I20260925 14:51:27.188089 12976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:27.189478 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34749 every 8 connection(s)
I20260925 14:51:27.191072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 12976
I20260925 14:51:27.191177 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:27.191365 392 external_mini_cluster.cc:1428] Setting key d61d1dd61fe43b0fc41cb3611f41bbe8
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:27.201419 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:27.190478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57211 (local address 127.0.98.62:34745)
0925 14:51:27.190524 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:27.190526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:27.190539 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:51:27.191143 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:27.191145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:27.191157 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:27.191190 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:27.192149 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.192905 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.194488 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.194801 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.194851 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:27.197629 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:27.197631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:27.197631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:27.197640 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:27.199668 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:27.200203 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:27.200204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:27.200204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:27.200271 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:27.200602 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:27.200603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:27.200603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:27.200751 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:27.200829 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:27.201226 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 14:51:27.201320 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:51:27.201592 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:27.190000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.98.3:57211)
0925 14:51:27.190281 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:27.190334 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:27.191004 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:27.191391 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:27.191395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:27.191651 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:27.192029 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:27.192036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.193028 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.193028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:27.194339 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:27.194342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.195017 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.195019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:27.195164 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:27.195822 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:27.195832 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:27.197494 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:27.199765 (+ 2271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:27.199772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:27.199775 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:27.200101 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:27.200356 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:27.200357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:27.200358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:27.200512 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:27.200847 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:27.200854 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:27.201048 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 14:51:27.201180 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 14:51:27.201353 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":106,"threads_started":1}
I20260925 14:51:27.202490 13105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34745
I20260925 14:51:27.202637 13105 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.202878 13105 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:27.203289 12621 ts_manager.cc:194] Registered new tserver with Master: 1525a7c599f245d3bfeed6527cabed42 (127.0.98.3:34749)
I20260925 14:51:27.203780 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57211
I20260925 14:51:27.214013 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:51:27.214879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 12702
I20260925 14:51:27.220890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 12839
I20260925 14:51:27.225536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 12976
I20260925 14:51:27.232012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 12589
I20260925 14:51:27.237496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--webserver_interface=127.0.98.62
--webserver_port=45249
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34745
--ranger_config_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:27.321164 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:27.321374 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:27.321396 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:27.322537 13108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:27.322579 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:27.322594 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:27.322606 13108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:27.322618 13108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:27.324163 13108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34745
--ranger_config_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45249
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:27.324345 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:27.324507 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:27.329416 13116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.329516 13114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.329396 13113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:27.329799 13108 server_base.cc:1034] running on GCE node
I20260925 14:51:27.330049 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:27.330344 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:27.331529 13108 hybrid_clock.cc:648] HybridClock initialized: now 1790347887331521 us; error 31 us; skew 500 ppm
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:27.334045 13108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:27.334553 13108 webserver.cc:460] Webserver started at http://127.0.98.62:45249/ using document root <none> and password file <none>
I20260925 14:51:27.334770 13108 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:27.334841 13108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:27.395064 13108 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:27.395946 13124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.396256 13108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:27.396418 13108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "0911cbc6d3454fd59e810f49571a1b2f"
format_stamp: "Formatted at 2026-09-25 14:51:25 on dist-test-slave-z2s5"
server_key: "dd22169a0c09df2f1a68af8764023f4f"
server_key_iv: "84ce6834681b61e774601015244e2658"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.396932 13108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:27.410125 13108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:27.414805 13108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:27.415037 13108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:27.420662 13108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34745
I20260925 14:51:27.420691 13176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34745 every 8 connection(s)
I20260925 14:51:27.421108 13108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:27.422446 13177 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:51:27.423134 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Bootstrap starting.
I20260925 14:51:27.425580 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13108
I20260925 14:51:27.425913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43641
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34745
--webserver_port=37737
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:27.432548 13177 log.cc:824] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:27.435007 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:51:27.435338 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Bootstrap complete.
I20260925 14:51:27.438957 13177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:27.439270 13177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0911cbc6d3454fd59e810f49571a1b2f, State: Initialized, Role: FOLLOWER
I20260925 14:51:27.439425 13177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:27.439487 13177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:27.439522 13177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:27.439566 13177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:51:27.440860 13177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:27.440965 13177 leader_election.cc:302] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0911cbc6d3454fd59e810f49571a1b2f; no voters:
I20260925 14:51:27.441118 13177 leader_election.cc:288] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:51:27.441223 13181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:51:27.441358 13177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:27.441426 13181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [term 2 LEADER]: Becoming Leader. State: Replica: 0911cbc6d3454fd59e810f49571a1b2f, State: Running, Role: LEADER
I20260925 14:51:27.441568 13181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } }
I20260925 14:51:27.441929 13182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0911cbc6d3454fd59e810f49571a1b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260925 14:51:27.442018 13182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:27.442160 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0911cbc6d3454fd59e810f49571a1b2f. Latest consensus state: current_term: 2 leader_uuid: "0911cbc6d3454fd59e810f49571a1b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0911cbc6d3454fd59e810f49571a1b2f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34745 } } }
I20260925 14:51:27.442214 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:27.442499 13190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:27.442785 13177 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:51:27.443140 13190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:27.445609 13190 catalog_manager.cc:1259] Loaded cluster ID: cfcc5e7a1f574d389f10d3ebdda0a89f
I20260925 14:51:27.445791 13190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:27.448199 13190 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:27.448518 13190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Loaded TSK: 0
I20260925 14:51:27.448858 13190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:51:27.520108 13179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:27.520340 13179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:27.520383 13179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:27.521749 13179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:27.521819 13179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:27.521865 13179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:27.523523 13179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43641
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37737
--tserver_master_addrs=127.0.98.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:27.523782 13179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:27.524068 13179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:27.530089 13215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.530115 13213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.530550 13212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:27.530661 13179 server_base.cc:1034] running on GCE node
I20260925 14:51:27.530923 13179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:27.531241 13179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:27.532407 13179 hybrid_clock.cc:648] HybridClock initialized: now 1790347887532392 us; error 27 us; skew 500 ppm
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:27.535212 13179 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:27.535653 13179 webserver.cc:460] Webserver started at http://127.0.98.1:37737/ using document root <none> and password file <none>
I20260925 14:51:27.535904 13179 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:27.535987 13179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:27.566936 13179 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 14:51:27.567873 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.568077 13179 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:27.568151 13179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "97c8be7f4f3844ada73b3edd0a0635ca"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "14972f6a904951ac7d4b8fe25d4521ea"
server_key_iv: "b462ccc9b4798f54e8eae2ef8d894609"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.568589 13179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:27.583797 13179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:27.586259 13179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:27.586522 13179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:27.586790 13179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:27.587170 13179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:27.587251 13179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.587352 13179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:27.587423 13179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.593634 13179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43641
I20260925 14:51:27.594180 13179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:27.596925 13338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43641 every 8 connection(s)
I20260925 14:51:27.602542 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13179
I20260925 14:51:27.602859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38257
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34745
--webserver_port=46025
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:27.611868 13340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:27.597611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.98.1:37607)
0925 14:51:27.598468 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:27.598513 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:27.599138 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:27.600143 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:27.600148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:27.600398 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:27.600672 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:27.600678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.602527 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.602528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:27.604206 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:27.604209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.605130 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.605131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:27.605264 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:27.605888 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:27.605898 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:27.607583 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:27.609922 (+ 2339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:27.609928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:27.609930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:27.610214 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:27.610553 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:27.610553 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:27.610554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:27.610687 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:27.611179 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:27.611186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:27.611345 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 14:51:27.611453 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 14:51:27.611614 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":812,"thread_start_us":84,"threads_started":1}
I20260925 14:51:27.612640 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:27.598026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37607 (local address 127.0.98.62:34745)
0925 14:51:27.599410 (+ 1384us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:27.599419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:27.599460 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:51:27.599603 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:27.599610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:27.599661 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:27.599846 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:27.600824 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.602351 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.604306 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.604916 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.605020 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:27.607737 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:27.607742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:27.607742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:27.607754 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:27.609825 (+ 2071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:27.610317 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:27.610320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:27.610321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:27.610464 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:27.610775 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:27.610776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:27.610777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:27.611075 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:27.611152 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:27.612304 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0925 14:51:27.612424 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1323,"thread_start_us":81,"threads_started":1}
I20260925 14:51:27.613758 13339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34745
I20260925 14:51:27.613853 13339 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.614051 13339 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
W20260925 14:51:27.614996 13339 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34745 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:51:27.615065 13339 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.615124 13339 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:27.615514 13339 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.615585 13339 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
W20260925 14:51:27.615947 13339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:51:27.729231 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:27.729497 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:27.729543 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:27.731033 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:27.731107 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:27.731153 13343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:27.732831 13343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38257
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46025
--tserver_master_addrs=127.0.98.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:27.733076 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:27.733340 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:27.738492 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.738490 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.738911 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:27.739457 13343 server_base.cc:1034] running on GCE node
I20260925 14:51:27.739657 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:27.739941 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:27.741103 13343 hybrid_clock.cc:648] HybridClock initialized: now 1790347887741092 us; error 30 us; skew 500 ppm
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:27.744004 13343 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:27.744506 13343 webserver.cc:460] Webserver started at http://127.0.98.2:46025/ using document root <none> and password file <none>
I20260925 14:51:27.744748 13343 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:27.744829 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:27.763713 13343 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:27.765100 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.765303 13343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:27.765404 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "98cef263518c4f399880f648060c8782"
format_stamp: "Formatted at 2026-09-25 14:51:26 on dist-test-slave-z2s5"
server_key: "d0c24b256b8a6ddff050bdef37c46b77"
server_key_iv: "0ba6a0a96b18073d4cea40157bb8c2f8"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.765923 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:27.790859 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:27.793583 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:27.793848 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:27.794175 13343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:27.794657 13343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:27.794766 13343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.794862 13343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:27.794924 13343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.802698 13343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38257
I20260925 14:51:27.802834 13472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38257 every 8 connection(s)
I20260925 14:51:27.803236 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:27.806282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13343
I20260925 14:51:27.806667 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46307/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34749
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34745
--webserver_port=37615
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:27.815711 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:27.804086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46999 (local address 127.0.98.62:34745)
0925 14:51:27.804145 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:27.804148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:27.804392 (+ 244us) server_negotiation.cc:407] Connection header received
0925 14:51:27.804977 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:27.804979 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:27.804995 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:27.805057 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:27.805682 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.806237 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.807901 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.808255 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.808316 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:27.811874 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:27.811876 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:27.811876 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:27.811886 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:27.813801 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:27.814377 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:27.814378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:27.814378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:27.814499 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:27.814824 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:27.814825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:27.814826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:27.814981 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:27.815029 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:27.815474 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 14:51:27.815571 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:51:27.816587 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:27.804100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.98.2:46999)
0925 14:51:27.804266 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:27.804311 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:27.804855 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:27.805136 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:27.805143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:27.805362 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:27.805573 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:27.805577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.806340 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.806341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:27.807780 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:27.807783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:27.808436 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:27.808437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:27.808600 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:27.809421 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:27.809433 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:27.811173 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:27.813906 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:27.813913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:27.813916 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:27.814278 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:27.814585 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:27.814586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:27.814586 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:27.814734 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:27.815081 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:27.815087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:27.815299 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 14:51:27.816105 (+ 806us) client_negotiation.cc:240] Negotiation successful
0925 14:51:27.816309 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260925 14:51:27.817432 13473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34745
I20260925 14:51:27.817540 13473 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.817747 13473 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
W20260925 14:51:27.818529 13473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34745 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:51:27.818599 13473 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.818666 13473 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:27.819048 13473 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:27.819115 13473 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
W20260925 14:51:27.819470 13473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:51:27.920125 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:27.920349 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:27.920388 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:27.921556 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:27.921594 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:27.921619 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:27.923188 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:46307/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37615
--tserver_master_addrs=127.0.98.62:34745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:27.923571 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:27.923858 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:27.933283 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.933375 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:27.933298 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:27.933820 13476 server_base.cc:1034] running on GCE node
I20260925 14:51:27.934026 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:27.934300 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:27.935446 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790347887935429 us; error 33 us; skew 500 ppm
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:27.941217 13476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:27.941856 13476 webserver.cc:460] Webserver started at http://127.0.98.3:37615/ using document root <none> and password file <none>
I20260925 14:51:27.942157 13476 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:27.942266 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 14:51:27 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:27.971614 13476 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260925 14:51:27.972896 13492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:27.973169 13476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:27.973261 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "1525a7c599f245d3bfeed6527cabed42"
format_stamp: "Formatted at 2026-09-25 14:51:27 on dist-test-slave-z2s5"
server_key: "fc3737fc35ce1125ee36994b356b91c2"
server_key_iv: "11c5218af8345a85c64174066d77ee60"
server_key_version: "kuduclusterkey@0"
I20260925 14:51:27.973718 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:28.001607 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:28.004231 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:28.004459 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:28.004714 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:28.005115 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:28.005196 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:28.005271 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:28.005321 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:28.015635 13476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34749
I20260925 14:51:28.016211 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:28.020332 13605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34749 every 8 connection(s)
I20260925 14:51:28.024748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13476
Sep 25 14:51:28 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:28.034335 13610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:28.025188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.98.62:34745)
0925 14:51:28.025339 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:28.025341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:28.025477 (+ 136us) server_negotiation.cc:407] Connection header received
0925 14:51:28.025579 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:28.025580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:28.025592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:28.025637 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:28.027214 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:28.027953 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:28.029013 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:28.029307 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:28.029355 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:28.030989 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:28.030991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:28.030991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:28.030999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:28.032866 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:28.033293 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:28.033295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:28.033295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:28.033359 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:28.033593 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:28.033594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:28.033594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:28.033727 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:28.033883 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:28.034112 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 14:51:28.034201 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":65,"threads_started":1}
I20260925 14:51:28.038760 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:28.021023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36105 (local address 127.0.98.62:34745)
0925 14:51:28.021077 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:28.021079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:28.021398 (+ 319us) server_negotiation.cc:407] Connection header received
0925 14:51:28.021938 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:28.021940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:28.021952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:28.022006 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:28.022694 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:28.023191 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:28.024188 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:28.024398 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:28.024488 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:28.035549 (+ 11061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:28.035550 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:28.035551 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:28.035559 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:28.036973 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:28.037555 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:28.037557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:28.037557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:28.037614 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:28.037930 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:28.037931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:28.037931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:28.038063 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:28.038108 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:28.038578 (+ 470us) server_negotiation.cc:299] Negotiation successful
0925 14:51:28.038663 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:51:28.039546 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:28.021041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34745 (local address 127.0.98.3:36105)
0925 14:51:28.021237 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:28.021287 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:28.021827 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:28.022091 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:28.022095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:28.022313 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:28.022594 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:28.022599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:28.023299 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:28.023300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:28.024090 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:28.024092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:28.024517 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:28.024518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:28.024662 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:28.025366 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:28.025378 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:28.034723 (+ 9345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:28.037086 (+ 2363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:28.037098 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:28.037100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:28.037458 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:28.037690 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:28.037691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:28.037691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:28.037842 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:28.038160 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:28.038166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:28.038376 (+ 210us) client_negotiation.cc:769] Sending connection context
0925 14:51:28.039104 (+ 728us) client_negotiation.cc:240] Negotiation successful
0925 14:51:28.039300 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
I20260925 14:51:28.040236 13606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34745
I20260925 14:51:28.040351 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:28.040546 13606 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
W20260925 14:51:28.041184 13606 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34745 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 14:51:28.041253 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:28.041329 13606 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:28.041689 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:28.041762 13606 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
W20260925 14:51:28.042093 13606 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:51:28.616154 13339 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:28.616331 13339 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:28.616873 13141 ts_manager.cc:194] Registered new tserver with Master: 97c8be7f4f3844ada73b3edd0a0635ca (127.0.98.1:43641)
I20260925 14:51:28.618062 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37607
Sep 25 14:51:28 dist-test-slave-z2s5 krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790347888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:28.819658 13473 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:28.819835 13473 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:28.820170 13141 ts_manager.cc:194] Registered new tserver with Master: 98cef263518c4f399880f648060c8782 (127.0.98.2:38257)
I20260925 14:51:28.820588 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46999
I20260925 14:51:29.042274 13606 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:29.042454 13606 heartbeater.cc:511] Master 127.0.98.62:34745 requested a full tablet report, sending...
I20260925 14:51:29.042820 13141 ts_manager.cc:194] Registered new tserver with Master: 1525a7c599f245d3bfeed6527cabed42 (127.0.98.3:34749)
I20260925 14:51:29.043388 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36105
I20260925 14:51:29.048691 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:51:29.058228 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.049735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.62:34745)
0925 14:51:29.049905 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.049907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.049944 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:51:29.050041 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.050042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.050055 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.050133 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:29.050933 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.051693 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.052737 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.053027 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.053095 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.054809 (+ 1714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:29.054810 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:29.054811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:29.054819 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:29.056719 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:29.057119 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:29.057120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:29.057121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:29.057185 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:29.057409 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:29.057410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:29.057410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:29.057548 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:29.057647 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.058034 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.058120 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260925 14:51:29.060421 13141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:29.063093 13141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:29.071630 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.066424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43641 (local address 127.0.0.1:59282)
0925 14:51:29.067047 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.067057 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:29.067101 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:29.067351 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:29.067354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:29.067357 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:29.067432 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.067434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.068737 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.068738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.070232 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.070235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.071332 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.071333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.071411 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.071425 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:51:29.071493 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:51:29.071542 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":82,"threads_started":1}
I20260925 14:51:29.072058 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.066518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.98.1:43641)
0925 14:51:29.067181 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.067187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.067196 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:29.067230 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.067232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.067248 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.067322 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:29.067503 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.068639 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.070329 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.071154 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.071224 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.071297 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.071939 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.071983 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":57,"threads_started":1}
I20260925 14:51:29.073014 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 71a3a77b83504b1ca9e312d0e9fcfb86 (DEFAULT_TABLE table=test-table [id=8058083cebd0483492cc3c0fb7149544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:29.073405 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a3a77b83504b1ca9e312d0e9fcfb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:29.076861 13640 tablet_bootstrap.cc:492] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: Bootstrap starting.
I20260925 14:51:29.077539 13640 tablet_bootstrap.cc:654] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:29.077908 13640 log.cc:824] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:29.079440 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.066160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.0.1:39050)
0925 14:51:29.066943 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.066982 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:29.067053 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:29.073079 (+ 6026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:29.073080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:29.073082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:29.073125 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.073126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.077104 (+ 3978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.077105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.078247 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.078250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.079223 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.079224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.079295 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.079306 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:51:29.079333 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 14:51:29.079365 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":76,"threads_started":1}
I20260925 14:51:29.079815 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.066275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.0.98.3:34749)
0925 14:51:29.072868 (+ 6593us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.072873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.072883 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:29.072951 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.072953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.072970 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.073197 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:29.073270 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.074326 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.078346 (+ 4020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.079123 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.079592 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.079686 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.079713 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.079759 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6557,"thread_start_us":67,"threads_started":1}
I20260925 14:51:29.079967 13640 tablet_bootstrap.cc:492] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: No bootstrap required, opened a new log
I20260925 14:51:29.080073 13640 ts_tablet_manager.cc:1397] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:51:29.080502 13521 tablet_service.cc:1467] Processing CreateTablet for tablet 71a3a77b83504b1ca9e312d0e9fcfb86 (DEFAULT_TABLE table=test-table [id=8058083cebd0483492cc3c0fb7149544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:29.080803 13521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a3a77b83504b1ca9e312d0e9fcfb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:29.084415 13642 tablet_bootstrap.cc:492] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: Bootstrap starting.
I20260925 14:51:29.084548 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.066781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.98.2:38257)
0925 14:51:29.067436 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.067440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.077200 (+ 9760us) server_negotiation.cc:407] Connection header received
0925 14:51:29.077246 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.077248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.077265 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.077341 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:29.077567 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.078839 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.083286 (+ 4447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.084050 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.084106 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.084177 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.084419 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.084465 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":48,"threads_started":1}
I20260925 14:51:29.084929 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.066610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38257 (local address 127.0.0.1:41196)
0925 14:51:29.072399 (+ 5789us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.072409 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:29.072445 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:29.077424 (+ 4979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:29.077458 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:29.077460 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:29.077489 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.077491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.078980 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.078981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.083159 (+ 4178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.083163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.084242 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.084243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.084321 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.084324 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:29.084802 (+ 478us) client_negotiation.cc:240] Negotiation successful
0925 14:51:29.084856 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5777,"thread_start_us":88,"threads_started":1}
I20260925 14:51:29.085083 13642 tablet_bootstrap.cc:654] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:29.085430 13642 log.cc:824] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:29.085743 13404 tablet_service.cc:1467] Processing CreateTablet for tablet 71a3a77b83504b1ca9e312d0e9fcfb86 (DEFAULT_TABLE table=test-table [id=8058083cebd0483492cc3c0fb7149544]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:29.086076 13404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71a3a77b83504b1ca9e312d0e9fcfb86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:29.086166 13642 tablet_bootstrap.cc:492] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: No bootstrap required, opened a new log
I20260925 14:51:29.086220 13642 ts_tablet_manager.cc:1397] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:29.090137 13642 raft_consensus.cc:348] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.090456 13642 raft_consensus.cc:374] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:29.090561 13642 raft_consensus.cc:729] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1525a7c599f245d3bfeed6527cabed42, State: Initialized, Role: FOLLOWER
I20260925 14:51:29.090757 13642 consensus_queue.cc:260] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.090727 13640 raft_consensus.cc:348] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.090924 13640 raft_consensus.cc:374] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:29.091014 13640 raft_consensus.cc:729] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c8be7f4f3844ada73b3edd0a0635ca, State: Initialized, Role: FOLLOWER
I20260925 14:51:29.091114 13642 ts_tablet_manager.cc:1428] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 14:51:29.091269 13606 heartbeater.cc:503] Master 127.0.98.62:34745 was elected leader, sending a full tablet report...
I20260925 14:51:29.091207 13640 consensus_queue.cc:260] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.091488 13640 ts_tablet_manager.cc:1428] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20260925 14:51:29.091609 13339 heartbeater.cc:503] Master 127.0.98.62:34745 was elected leader, sending a full tablet report...
I20260925 14:51:29.092216 13644 tablet_bootstrap.cc:492] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782: Bootstrap starting.
I20260925 14:51:29.092836 13644 tablet_bootstrap.cc:654] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:29.093130 13644 log.cc:824] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:29.094000 13644 tablet_bootstrap.cc:492] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782: No bootstrap required, opened a new log
I20260925 14:51:29.094115 13644 ts_tablet_manager.cc:1397] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:29.096931 13644 raft_consensus.cc:348] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.097119 13644 raft_consensus.cc:374] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:29.097154 13644 raft_consensus.cc:729] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98cef263518c4f399880f648060c8782, State: Initialized, Role: FOLLOWER
I20260925 14:51:29.097311 13644 consensus_queue.cc:260] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.097589 13644 ts_tablet_manager.cc:1428] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:29.097620 13473 heartbeater.cc:503] Master 127.0.98.62:34745 was elected leader, sending a full tablet report...
W20260925 14:51:29.104732 13342 tablet.cc:2369] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:29.108086 13645 raft_consensus.cc:482] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:29.108306 13645 raft_consensus.cc:504] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.108739 13645 leader_election.cc:288] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 98cef263518c4f399880f648060c8782 (127.0.98.2:38257), 97c8be7f4f3844ada73b3edd0a0635ca (127.0.98.1:43641)
I20260925 14:51:29.113476 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.109424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60301 (local address 127.0.98.1:43641)
0925 14:51:29.109470 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.109472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.109591 (+ 119us) server_negotiation.cc:407] Connection header received
0925 14:51:29.109700 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.109701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.109713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.109746 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:29.109973 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.110762 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.112449 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.113037 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.113089 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.113101 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.113353 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.113398 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:51:29.114357 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.109159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43641 (local address 127.0.98.3:60301)
0925 14:51:29.109528 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.109538 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:29.109617 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:29.109818 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:29.109819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:29.109820 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:29.109891 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.109893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.110871 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.110872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.112323 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.112327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.113168 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.113169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.113261 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.113265 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:29.113796 (+ 531us) client_negotiation.cc:240] Negotiation successful
0925 14:51:29.114259 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":179,"thread_start_us":84,"threads_started":1}
I20260925 14:51:29.114881 13291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a3a77b83504b1ca9e312d0e9fcfb86" candidate_uuid: "1525a7c599f245d3bfeed6527cabed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" is_pre_election: true
I20260925 14:51:29.115064 13291 raft_consensus.cc:2454] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1525a7c599f245d3bfeed6527cabed42 in term 0.
I20260925 14:51:29.115877 13494 leader_election.cc:302] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1525a7c599f245d3bfeed6527cabed42, 97c8be7f4f3844ada73b3edd0a0635ca; no voters:
I20260925 14:51:29.116029 13645 raft_consensus.cc:2781] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:29.116070 13645 raft_consensus.cc:482] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:29.116096 13645 raft_consensus.cc:3037] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:29.116988 13645 raft_consensus.cc:504] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.117129 13645 leader_election.cc:288] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [CANDIDATE]: Term 1 election: Requested vote from peers 98cef263518c4f399880f648060c8782 (127.0.98.2:38257), 97c8be7f4f3844ada73b3edd0a0635ca (127.0.98.1:43641)
I20260925 14:51:29.117588 13291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a3a77b83504b1ca9e312d0e9fcfb86" candidate_uuid: "1525a7c599f245d3bfeed6527cabed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c8be7f4f3844ada73b3edd0a0635ca"
I20260925 14:51:29.117677 13291 raft_consensus.cc:3037] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:29.118712 13291 raft_consensus.cc:2454] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1525a7c599f245d3bfeed6527cabed42 in term 1.
I20260925 14:51:29.118938 13494 leader_election.cc:302] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1525a7c599f245d3bfeed6527cabed42, 97c8be7f4f3844ada73b3edd0a0635ca; no voters:
I20260925 14:51:29.119064 13645 raft_consensus.cc:2781] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:29.119261 13645 raft_consensus.cc:686] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 1 LEADER]: Becoming Leader. State: Replica: 1525a7c599f245d3bfeed6527cabed42, State: Running, Role: LEADER
I20260925 14:51:29.119356 13645 consensus_queue.cc:237] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } }
I20260925 14:51:29.120292 13141 catalog_manager.cc:5495] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1525a7c599f245d3bfeed6527cabed42 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1525a7c599f245d3bfeed6527cabed42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1525a7c599f245d3bfeed6527cabed42" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34749 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 } health_report { overall_health: UNKNOWN } } }
I20260925 14:51:29.129081 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.108915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38257 (local address 127.0.98.3:48105)
0925 14:51:29.109110 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.109124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:29.109234 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:29.109534 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:29.109535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:29.109538 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:29.109630 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.109632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.110856 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.110856 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.126142 (+ 15286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:29.126147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.128763 (+ 2616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.128764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:29.128854 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.128858 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:29.128914 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:51:29.128967 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":36,"thread_start_us":112,"threads_started":1}
I20260925 14:51:29.129088 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.109315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48105 (local address 127.0.98.2:38257)
0925 14:51:29.109374 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.109376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.109385 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:29.109412 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.109413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.109428 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.109480 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:29.109715 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.110483 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.127963 (+ 17480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.128605 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.128661 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.128673 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.128952 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.129014 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:51:29.129702 13426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a3a77b83504b1ca9e312d0e9fcfb86" candidate_uuid: "1525a7c599f245d3bfeed6527cabed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cef263518c4f399880f648060c8782"
I20260925 14:51:29.129663 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71a3a77b83504b1ca9e312d0e9fcfb86" candidate_uuid: "1525a7c599f245d3bfeed6527cabed42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98cef263518c4f399880f648060c8782" is_pre_election: true
I20260925 14:51:29.129820 13426 raft_consensus.cc:3037] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:29.130797 13426 raft_consensus.cc:2454] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1525a7c599f245d3bfeed6527cabed42 in term 1.
I20260925 14:51:29.134768 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.130934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39052 (local address 127.0.98.3:34749)
0925 14:51:29.130981 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:29.130983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:29.130991 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:29.131030 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:29.131031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:29.131056 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:29.131148 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:29.131476 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.132220 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.133553 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:29.133853 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:29.133918 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:29.134130 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:29.134273 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:29.134354 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:29.134665 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 14:51:29.134688 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:51:29.136873 13291 raft_consensus.cc:1261] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 97c8be7f4f3844ada73b3edd0a0635ca [term 1 FOLLOWER]: Refusing update from remote peer 1525a7c599f245d3bfeed6527cabed42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:29.137282 13651 consensus_queue.cc:1035] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c8be7f4f3844ada73b3edd0a0635ca" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:29.139761 13426 raft_consensus.cc:1261] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 98cef263518c4f399880f648060c8782 [term 1 FOLLOWER]: Refusing update from remote peer 1525a7c599f245d3bfeed6527cabed42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:29.140044 13651 consensus_queue.cc:1035] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98cef263518c4f399880f648060c8782" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Sep 25 14:51:29 dist-test-slave-z2s5 krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260925 14:51:29.144690 13654 mvcc.cc:205] Tried to move back new op lower bound from 7333264953901322240 to 7333264953833459712. Current Snapshot: MvccSnapshot[applied={T|T < 7333264953901322240}]
I20260925 14:51:29.148777 13655 mvcc.cc:205] Tried to move back new op lower bound from 7333264953901322240 to 7333264953833459712. Current Snapshot: MvccSnapshot[applied={T|T < 7333264953901322240}]
I20260925 14:51:29.150933 13657 mvcc.cc:205] Tried to move back new op lower bound from 7333264953901322240 to 7333264953833459712. Current Snapshot: MvccSnapshot[applied={T|T < 7333264953901322240}]
I20260925 14:51:29.153781 13141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:29.153925 13141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:29.154760 13141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0911cbc6d3454fd59e810f49571a1b2f: Sending DeleteTablet for 3 replicas of tablet 71a3a77b83504b1ca9e312d0e9fcfb86
I20260925 14:51:29.155210 13521 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a3a77b83504b1ca9e312d0e9fcfb86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39050
I20260925 14:51:29.155220 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a3a77b83504b1ca9e312d0e9fcfb86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59282
I20260925 14:51:29.155483 13404 tablet_service.cc:1514] Processing DeleteTablet for tablet 71a3a77b83504b1ca9e312d0e9fcfb86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41196
I20260925 14:51:29.155718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13179
W20260925 14:51:29.163620 13126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43641 (error 108)
I20260925 14:51:29.164299 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13343
I20260925 14:51:29.170816 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.170678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43641 (local address 127.0.0.1:59292)
0925 14:51:29.170736 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.170775 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43641: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 14:51:29.171079 13126 catalog_manager.cc:4584] TS 97c8be7f4f3844ada73b3edd0a0635ca (127.0.98.1:43641): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71a3a77b83504b1ca9e312d0e9fcfb86: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43641: connect: Connection refused (error 111)
I20260925 14:51:29.175208 13664 tablet_replica.cc:330] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42: stopping tablet replica
I20260925 14:51:29.175376 13664 raft_consensus.cc:2229] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:29.175529 13664 raft_consensus.cc:2258] T 71a3a77b83504b1ca9e312d0e9fcfb86 P 1525a7c599f245d3bfeed6527cabed42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:29.177582 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:29.177453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38257 (local address 127.0.0.1:41200)
0925 14:51:29.177516 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.177543 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:51:29.177714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13476
W20260925 14:51:29.184379 13638 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:29.184087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34749 (local address 127.0.0.1:39066)
0925 14:51:29.184132 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:29.184143 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:29.184182 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:29.184316 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:34749: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 14:51:29.184974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13108
I20260925 14:51:29.191735 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:51:29.536466 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:51:29.536559 392 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:51:29.881385 392 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:51:29.881 UTC [12175] LOG: received smart shutdown request
2026-09-25 14:51:29.885 UTC [12175] LOG: background worker "logical replication launcher" (PID 12184) exited with exit code 1
2026-09-25 14:51:29.885 UTC [12179] LOG: shutting down
2026-09-25 14:51:30.080 UTC [12175] LOG: database system is shut down
2026-09-25T14:51:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49138 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:30 dist-test-slave-z2s5 krb5kdc[13675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:30 dist-test-slave-z2s5 krb5kdc[13675](info): set up 2 sockets
Sep 25 14:51:30 dist-test-slave-z2s5 krb5kdc[13675](info): commencing operation
krb5kdc: starting...
W20260925 14:51:31.238511 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:31.356676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41907,127.0.98.61:43133,127.0.98.60:41361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:31.436841 13688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:31.437050 13688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:31.437072 13688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:31.438127 13688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:31.438163 13688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:31.438177 13688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:31.438190 13688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:31.438202 13688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:31.439674 13688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41907,127.0.98.61:43133,127.0.98.60:41361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:31.439874 13688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:31.440038 13688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:31.444854 13693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:31.444859 13696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:31.445030 13688 server_base.cc:1034] running on GCE node
W20260925 14:51:31.444860 13694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:31.445356 13688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:31.445655 13688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:31.446816 13688 hybrid_clock.cc:648] HybridClock initialized: now 1790347891446807 us; error 52 us; skew 500 ppm
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:31.453814 13688 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:31.454264 13688 webserver.cc:460] Webserver started at http://127.0.98.62:33677/ using document root <none> and password file <none>
I20260925 14:51:31.454546 13688 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:31.454615 13688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:31.454772 13688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:31.455889 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "82720c44ce3a4ec38aa26da79be4e951"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "a723051c1fdbf496f5e6ad91be46477a"
server_key_iv: "c9d7ea2c979eb0ade0a8dd1f86e892d9"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.456257 13688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "82720c44ce3a4ec38aa26da79be4e951"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "a723051c1fdbf496f5e6ad91be46477a"
server_key_iv: "c9d7ea2c979eb0ade0a8dd1f86e892d9"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.457566 13688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:31.458313 13703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:31.458510 13688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:31.458608 13688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "82720c44ce3a4ec38aa26da79be4e951"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "a723051c1fdbf496f5e6ad91be46477a"
server_key_iv: "c9d7ea2c979eb0ade0a8dd1f86e892d9"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.458712 13688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:31.473035 13688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:31.475173 13688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:31.475343 13688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:31.479475 13688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41907
I20260925 14:51:31.479511 13755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41907 every 8 connection(s)
I20260925 14:51:31.479872 13688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:31.480871 13756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:31.481549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13688
I20260925 14:51:31.481652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:31.481949 392 external_mini_cluster.cc:1428] Setting key 8d092f3635f1debcdfcc87bb946c6d50
I20260925 14:51:31.483809 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:31.497599 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.484606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.62:41907)
0925 14:51:31.484757 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.484773 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.484935 (+ 162us) server_negotiation.cc:407] Connection header received
0925 14:51:31.485496 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.485498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.485514 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.485609 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.486269 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.487734 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.488662 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.489177 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.489311 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.494144 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.494152 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.494152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.494159 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.495954 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.496427 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.496430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.496430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.496517 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.496785 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.496786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.496786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.497010 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.497075 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.497316 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.497425 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 14:51:31.497696 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.484586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41907 (local address 127.0.0.1:55042)
0925 14:51:31.484821 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.484850 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.485322 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.485614 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.485617 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.485903 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.486062 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.486067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.487861 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.487862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.488568 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.488570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.489272 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.489273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.489344 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.489913 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.489922 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.494058 (+ 4136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.496046 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.496053 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.496056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.496329 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.496588 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.496589 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.496589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.496708 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.497088 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.497096 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.497179 (+ 83us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.497291 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.497519 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":60,"thread_start_us":85,"threads_started":1}
I20260925 14:51:31.498548 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:51:31.498922 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.498783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56728)
0925 14:51:31.498838 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.498889 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:51:31.499073 13707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
I20260925 14:51:31.499507 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.499400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56742)
0925 14:51:31.499453 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.499485 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:51:31.499662 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43133: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 14:51:31.508916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43133
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41907,127.0.98.61:43133,127.0.98.60:41361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:31.548914 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } attempt: 1
I20260925 14:51:31.549499 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.549339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56754)
0925 14:51:31.549415 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.549463 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:51:31.549876 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.549788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56768)
0925 14:51:31.549831 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.549857 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:51:31.549969 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43133: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
W20260925 14:51:31.589273 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:31.589480 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:31.589502 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:31.590615 13763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:31.590655 13763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:31.590670 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:31.590687 13763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:31.590700 13763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:31.592116 13763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41907,127.0.98.61:43133,127.0.98.60:41361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:31.592300 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:31.592464 13763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:31.597172 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:31.597167 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:31.597167 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:31.597427 13763 server_base.cc:1034] running on GCE node
I20260925 14:51:31.597687 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:31.597963 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:31.599105 13763 hybrid_clock.cc:648] HybridClock initialized: now 1790347891599082 us; error 35 us; skew 500 ppm
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:31.601630 13763 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 14:51:31.602062 13763 webserver.cc:460] Webserver started at http://127.0.98.61:35841/ using document root <none> and password file <none>
I20260925 14:51:31.602236 13763 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:31.602275 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:31.602360 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:31.603435 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data/instance:
uuid: "059e4d3ba4a845c59403a78e22a36f7b"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "d64556871530a2b80d25fc93790907c4"
server_key_iv: "2c36e61529c30e4176fe4a11e1ec7be5"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.603725 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal/instance:
uuid: "059e4d3ba4a845c59403a78e22a36f7b"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "d64556871530a2b80d25fc93790907c4"
server_key_iv: "2c36e61529c30e4176fe4a11e1ec7be5"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.604914 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:51:31.605571 13778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:31.605768 13763 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:31.605867 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal
uuid: "059e4d3ba4a845c59403a78e22a36f7b"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "d64556871530a2b80d25fc93790907c4"
server_key_iv: "2c36e61529c30e4176fe4a11e1ec7be5"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.605947 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:31.618105 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } attempt: 2
I20260925 14:51:31.618562 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.618449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56778)
0925 14:51:31.618500 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.618535 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 14:51:31.618903 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.618806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56794)
0925 14:51:31.618857 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.618885 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:51:31.618993 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43133: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43133: connect: Connection refused (error 111)
I20260925 14:51:31.623483 13763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:31.625825 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:31.625993 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:31.630043 13763 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43133
I20260925 14:51:31.630074 13830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43133 every 8 connection(s)
I20260925 14:51:31.630439 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:51:31.631333 13831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:31.632828 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:51:31.633545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13763
I20260925 14:51:31.633661 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-1/wal/instance
I20260925 14:51:31.633841 392 external_mini_cluster.cc:1428] Setting key fc6f7cad3f1a8892270fd6b953232dee
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260925 14:51:31.641813 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.633497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55048 (local address 127.0.98.62:41907)
0925 14:51:31.633566 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.633568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.633757 (+ 189us) server_negotiation.cc:407] Connection header received
0925 14:51:31.634278 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.634280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.634293 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.634349 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.635010 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.635509 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.636507 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.636721 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.636811 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.638943 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.638944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.638945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.638953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.640390 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.640896 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.640897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.640898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.640954 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.641188 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.641189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.641189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.641327 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.641400 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.641679 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.641747 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:51:31.642102 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.633460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41907 (local address 127.0.0.1:55048)
0925 14:51:31.633600 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.633640 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.634175 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.634456 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.634462 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.634708 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.634909 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.634915 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.635619 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.635620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.636416 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.636418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.636819 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.636819 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.636946 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.637462 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.637471 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.638839 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.640498 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.640504 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.640506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.640814 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.641021 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.641021 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.641021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.641116 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.641404 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.641409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.641572 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.641660 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.641779 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260925 14:51:31.642808 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:51:31.650779 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.643124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56802)
0925 14:51:31.643347 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.643359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.643403 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.643551 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.643552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.643683 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.643725 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.643726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.644605 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.644606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.645097 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.645099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.645577 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.645577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.645642 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.646074 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.646078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.647535 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.649529 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.649530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.649531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.649698 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.649958 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.649958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.649958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.650007 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.650369 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.650370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.650559 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.650634 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.650711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214}
I20260925 14:51:31.650897 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.643202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.0.98.61:43133)
0925 14:51:31.643356 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.643380 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.643404 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:51:31.643466 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.643468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.643481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.643541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.643810 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.644512 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.645184 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.645490 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.645577 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.647628 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.647636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.647636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.647644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.649441 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.649799 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.649803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.649804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.649878 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.650073 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.650074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.650075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.650273 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.650415 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.650699 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.650790 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":101,"threads_started":1}
I20260925 14:51:31.651592 13831 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:51:31.651938 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.651814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39418)
0925 14:51:31.651858 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.651920 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:51:31.652097 13782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
I20260925 14:51:31.652544 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.652405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39434)
0925 14:51:31.652493 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.652524 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 14:51:31.652701 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 14:51:31.659062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41361
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41907,127.0.98.61:43133,127.0.98.60:41361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:31.701957 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } attempt: 1
I20260925 14:51:31.702536 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.702335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39448)
0925 14:51:31.702428 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.702495 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:51:31.702934 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.702832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39458)
0925 14:51:31.702885 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.702914 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:51:31.703060 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
I20260925 14:51:31.710126 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } attempt: 3
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:51:31.717715 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.710513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56814)
0925 14:51:31.710559 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.710572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.710647 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.710794 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.710795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.710939 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.711027 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.711029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.711996 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.711996 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.712480 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.712481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.712984 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.712985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.713046 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.713547 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.713551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.714928 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.716686 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.716687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.716687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.716849 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.717106 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.717106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.717106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.717158 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.717469 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.717470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.717511 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.717567 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.717644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:51:31.717796 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.710576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.98.61:43133)
0925 14:51:31.710641 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.710643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.710655 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:31.710711 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.710712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.710728 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.710782 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.711104 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.711904 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.712562 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.712894 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.712966 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.715033 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.715035 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.715035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.715043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.716585 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.716947 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.716949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.716949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.717025 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.717219 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.717220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.717220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.717374 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.717450 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.717613 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.717705 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:51:31.718454 13756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:51:31.718794 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.718694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39460)
0925 14:51:31.718741 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.718772 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 14:51:31.719113 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.719033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39474)
0925 14:51:31.719074 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.719097 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:51:31.719224 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
W20260925 14:51:31.743373 13838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:31.743585 13838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:31.743608 13838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:31.744689 13838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:31.744724 13838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:31.744737 13838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:31.744750 13838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:31.744761 13838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:31.746307 13838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41907,127.0.98.61:43133,127.0.98.60:41361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:31.746587 13838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:31.746774 13838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:51:31.750381 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } attempt: 1
I20260925 14:51:31.750866 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.750720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39490)
0925 14:51:31.750785 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.750822 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:51:31.751389 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.751230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39504)
0925 14:51:31.751301 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.751345 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:51:31.751345 13843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:31.751425 13844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:31.751379 13846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:31.751672 13838 server_base.cc:1034] running on GCE node
W20260925 14:51:31.751735 13756 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
I20260925 14:51:31.751935 13838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:31.752267 13838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:31.753402 13838 hybrid_clock.cc:648] HybridClock initialized: now 1790347891753372 us; error 48 us; skew 500 ppm
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:31.755915 13838 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 14:51:31.756371 13838 webserver.cc:460] Webserver started at http://127.0.98.60:39997/ using document root <none> and password file <none>
I20260925 14:51:31.756574 13838 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:31.756616 13838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:31.756704 13838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:31.757720 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data/instance:
uuid: "040966ee543b49fa895dd0dc3efb1b73"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "08c2b27559d425432d7858e63f0057a3"
server_key_iv: "a3524ffc0da7e5aeba6de357cddc7086"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.758031 13838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal/instance:
uuid: "040966ee543b49fa895dd0dc3efb1b73"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "08c2b27559d425432d7858e63f0057a3"
server_key_iv: "a3524ffc0da7e5aeba6de357cddc7086"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.759470 13838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:31.760358 13853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:31.760574 13838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:31.760689 13838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal
uuid: "040966ee543b49fa895dd0dc3efb1b73"
format_stamp: "Formatted at 2026-09-25 14:51:31 on dist-test-slave-z2s5"
server_key: "08c2b27559d425432d7858e63f0057a3"
server_key_iv: "a3524ffc0da7e5aeba6de357cddc7086"
server_key_version: "encryptionkey@0"
I20260925 14:51:31.760774 13838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:31.771247 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } attempt: 2
I20260925 14:51:31.771781 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.771630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39510)
0925 14:51:31.771699 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.771746 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:51:31.772233 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.772106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39514)
0925 14:51:31.772174 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.772207 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 14:51:31.772363 13831 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41361: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41361: connect: Connection refused (error 111)
I20260925 14:51:31.793183 13838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:31.795768 13838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:31.795972 13838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:31.800030 13838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41361
I20260925 14:51:31.800063 13905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41361 every 8 connection(s)
I20260925 14:51:31.800433 13838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:51:31.801318 13906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:31.802996 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:51:31.804073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13838
I20260925 14:51:31.804165 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790347739045519-392-0/minicluster-data/master-2/wal/instance
I20260925 14:51:31.804352 392 external_mini_cluster.cc:1428] Setting key 22e8985f73fe0f69075272cc152a7d89
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:51:31.813495 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.803720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.0.98.62:41907)
0925 14:51:31.803801 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.803803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.804026 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:51:31.804716 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.804718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.804731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.804798 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.805532 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.806176 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.807257 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.807558 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.807631 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.810091 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.810093 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.810093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.810101 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.811758 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.812482 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.812483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.812483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.812535 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.812804 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.812805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.812805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.812916 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.813001 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.813304 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.813385 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:51:31.813678 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.803671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41907 (local address 127.0.0.1:55060)
0925 14:51:31.803861 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.803927 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.804582 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.804940 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.804945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.805283 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.805440 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.805444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.806258 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.806259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.807179 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.807181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.807633 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.807633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.807773 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.808353 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.808361 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.810004 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.812012 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.812019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.812021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.812388 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.812604 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.812605 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.812606 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.812737 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.813007 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.813014 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.813159 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.813271 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.813426 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260925 14:51:31.814272 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.804713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.0.98.60:41361)
0925 14:51:31.804918 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.804922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.804940 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:51:31.804957 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.804959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.804983 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.805070 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.805722 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.806762 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.807610 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.808032 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.808084 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.810759 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.810766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.810766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.810773 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.812907 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.813235 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.813238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.813239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.813303 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.813502 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.813503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.813504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.813688 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.813833 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.814094 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.814176 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260925 14:51:31.814268 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:31.821930 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.814536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43133 (local address 127.0.0.1:56820)
0925 14:51:31.814710 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.814720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.814801 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.814964 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.814965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.815123 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.815181 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.815183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.816122 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.816123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.816760 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.816762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.817123 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.817124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.817206 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.817651 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.817656 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.819095 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.820883 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.820884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.820884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.821075 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.821256 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.821257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.821257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.821320 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.821609 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.821610 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.821647 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.821700 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.821817 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 14:51:31.821940 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.814613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.0.98.61:43133)
0925 14:51:31.814657 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.814658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.814784 (+ 126us) server_negotiation.cc:407] Connection header received
0925 14:51:31.814855 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.814856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.814866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.814959 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.815250 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.816046 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.816858 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.817054 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.817106 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.819191 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.819192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.819193 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.819199 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.820732 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.821147 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.821147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.821148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.821196 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.821381 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.821381 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.821382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.821537 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.821610 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.821749 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.821825 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:51:31.822546 13906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:51:31.822844 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.815206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55074 (local address 127.0.98.62:41907)
0925 14:51:31.815240 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.815241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.815299 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:51:31.815364 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.815365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.815372 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.815412 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.815963 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.816384 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.817201 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.817368 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.817445 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.819646 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.819648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.819649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.819657 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.821370 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.821755 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.821756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.821757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.821813 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.822231 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.822232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.822232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.822376 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.822603 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.822655 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.822735 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:51:31.826956 13756 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } attempt: 2
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:51:31.830430 13908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.822797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39534)
0925 14:51:31.822933 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.822942 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.822986 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.823106 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.823107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.823259 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.823291 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.823293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.824164 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.824165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.824942 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.824945 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.825431 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.825431 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.825475 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.826012 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.826016 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.827311 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.829421 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.829422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.829422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.829607 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.829764 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.829765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.829765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.829806 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.830069 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.830070 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.830108 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.830178 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.830276 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 14:51:31.830490 13910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.822872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.0.98.60:41361)
0925 14:51:31.822912 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.822914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.822973 (+ 59us) server_negotiation.cc:407] Connection header received
0925 14:51:31.823039 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.823040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.823050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.823103 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.823362 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.824082 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.825043 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.825351 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.825434 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.827403 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.827404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.827404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.827412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.829335 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.829671 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.829671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.829672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.829714 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.829872 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.829873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.829873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.829992 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.830077 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.830205 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.830322 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":21}
I20260925 14:51:31.832088 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73: Bootstrap starting.
I20260925 14:51:31.832752 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.823533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.0.98.61:43133)
0925 14:51:31.823566 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.823567 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.823597 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:51:31.823659 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.823659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.823665 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.823708 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.824229 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.824912 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.826001 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.826285 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.826357 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.829100 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.829102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.829102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.829108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.831231 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.831938 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.831939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.831940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.832001 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.832220 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.832221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.832221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.832347 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.832553 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.832596 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.832670 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:51:31.832887 13906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73: Neither blocks nor log segments found. Creating new log.
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:51:31.833242 13906 log.cc:824] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:31.834714 13906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73: No bootstrap required, opened a new log
I20260925 14:51:31.834857 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:51:31.836581 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.827211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39542)
0925 14:51:31.827252 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.827263 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.827301 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.828362 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.828364 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.828507 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.828541 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.828543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.830156 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.830157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.830978 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.830981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.831424 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.831425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.831487 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.831890 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.831896 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.833406 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.835394 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.835395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.835396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.835629 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.835854 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.835855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.835856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.835933 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.836221 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.836223 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.836275 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.836351 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.836461 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:51:31.836594 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.827269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39542 (local address 127.0.98.60:41361)
0925 14:51:31.828218 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.828220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.828226 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:51:31.828266 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.828267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.828277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.828342 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.828626 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.829842 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.831066 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.831340 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.831415 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.833489 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.833490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.833490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.833498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.835293 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.835716 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.835717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.835717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.835776 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.835994 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.835995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.835995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.836142 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.836213 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.836392 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.836474 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":108,"threads_started":1}
I20260925 14:51:31.838577 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: Bootstrap starting.
I20260925 14:51:31.838574 13906 raft_consensus.cc:348] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.838755 13906 raft_consensus.cc:374] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:31.838798 13906 raft_consensus.cc:729] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 040966ee543b49fa895dd0dc3efb1b73, State: Initialized, Role: FOLLOWER
I20260925 14:51:31.838937 13906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.839243 13906 sys_catalog.cc:564] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:31.839329 13756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:31.839638 13756 log.cc:824] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:31.841288 13756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: No bootstrap required, opened a new log
W20260925 14:51:31.842690 13932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:51:31.842788 13932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:51:31.841404 13838 master_runner.cc:186] Error getting master registration for 127.0.98.62:41907: OK, instance_id { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" instance_seqno: 1790347891478196 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:51:31.844308 13838 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:51:31.840569 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.845441 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:51:31.846016 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.836391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.0.98.62:41907)
0925 14:51:31.836430 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.836432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.836709 (+ 277us) server_negotiation.cc:407] Connection header received
0925 14:51:31.836814 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.836815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.836825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.836855 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.837718 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.838464 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.840026 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.840313 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.840402 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.842468 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.842470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.842470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.842479 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.844425 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.844806 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.844807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.844807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.844863 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.845078 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.845079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.845079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.845205 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.845297 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.845817 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.845912 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:51:31.846746 13756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.846882 13756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:31.846917 13756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82720c44ce3a4ec38aa26da79be4e951, State: Initialized, Role: FOLLOWER
I20260925 14:51:31.847054 13756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.847373 13756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:31.848026 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.837211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.98.60:41361)
0925 14:51:31.837248 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.837249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.840689 (+ 3440us) server_negotiation.cc:407] Connection header received
0925 14:51:31.840793 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.840794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.840804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.840835 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.841504 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.842250 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.843237 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.843561 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.843607 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.845040 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.845042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.845042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.845049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.846749 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.847139 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.847140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.847141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.847195 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.847392 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.847393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.847393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.847536 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.847632 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.847850 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.847930 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:51:31.848573 13835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.836913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.0.98.61:43133)
0925 14:51:31.836949 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.836950 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.837383 (+ 433us) server_negotiation.cc:407] Connection header received
0925 14:51:31.837428 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.837429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.837439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.837497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.838043 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.838655 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.839793 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.840113 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.840157 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.845162 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.845164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.845164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.845171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.847008 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.847521 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.847523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.847523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.847575 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.847778 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.847778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.847779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.847903 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.847977 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.848391 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.848470 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:51:31.850540 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.850651 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:51:31.851640 13945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:51:31.851696 13945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:51:31.849992 13688 master_runner.cc:186] Error getting master registration for 127.0.98.61:43133: OK, instance_id { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" instance_seqno: 1790347891628872 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:51:31.852250 13688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:51:31.853186 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:31.853295 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.853863 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040966ee543b49fa895dd0dc3efb1b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82720c44ce3a4ec38aa26da79be4e951" is_pre_election: true
I20260925 14:51:31.854018 13730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 040966ee543b49fa895dd0dc3efb1b73 in term 0.
I20260925 14:51:31.853869 13920 leader_election.cc:288] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82720c44ce3a4ec38aa26da79be4e951 (127.0.98.62:41907), 059e4d3ba4a845c59403a78e22a36f7b (127.0.98.61:43133)
I20260925 14:51:31.854375 13855 leader_election.cc:302] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040966ee543b49fa895dd0dc3efb1b73, 82720c44ce3a4ec38aa26da79be4e951; no voters:
I20260925 14:51:31.854539 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:31.854652 13920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:31.854707 13920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:31.855621 13920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.855919 13920 leader_election.cc:288] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [CANDIDATE]: Term 1 election: Requested vote from peers 82720c44ce3a4ec38aa26da79be4e951 (127.0.98.62:41907), 059e4d3ba4a845c59403a78e22a36f7b (127.0.98.61:43133)
I20260925 14:51:31.855909 13730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040966ee543b49fa895dd0dc3efb1b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82720c44ce3a4ec38aa26da79be4e951"
I20260925 14:51:31.855981 13730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:31.856834 13730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040966ee543b49fa895dd0dc3efb1b73 in term 1.
I20260925 14:51:31.857023 13855 leader_election.cc:302] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 040966ee543b49fa895dd0dc3efb1b73, 82720c44ce3a4ec38aa26da79be4e951; no voters:
I20260925 14:51:31.857118 13920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:31.857265 13920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [term 1 LEADER]: Becoming Leader. State: Replica: 040966ee543b49fa895dd0dc3efb1b73, State: Running, Role: LEADER
I20260925 14:51:31.857363 13920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.857810 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040966ee543b49fa895dd0dc3efb1b73. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.858002 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:31.858302 13949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:31.858771 13949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:31.860499 13730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [term 1 FOLLOWER]: Refusing update from remote peer 040966ee543b49fa895dd0dc3efb1b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:31.860826 13947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:31.862272 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040966ee543b49fa895dd0dc3efb1b73. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.862423 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:51:31.863418 13947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.863508 13947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:31.863493 13831 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } attempt: 3
I20260925 14:51:31.864173 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.864259 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:51:31.864979 13951 mvcc.cc:205] Tried to move back new op lower bound from 7333264965058023424 to 7333264965048463360. Current Snapshot: MvccSnapshot[applied={T|T < 7333264965058023424}]
I20260925 14:51:31.865463 13949 catalog_manager.cc:1347] Generated new cluster ID: 2975a6d3ba3241f9b75e6462177a058e
I20260925 14:51:31.866508 13949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:51:31 dist-test-slave-z2s5 krb5kdc[13675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:51:31.871950 13833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.863751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41361 (local address 127.0.0.1:39556)
0925 14:51:31.863802 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:31.863814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:31.863853 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:31.864397 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:31.864399 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:31.864586 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:31.864635 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.864637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.865513 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.865515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.866355 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:31.866357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.866856 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.866857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:31.866926 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.867376 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:31.867382 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:31.869000 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:31.870912 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.870913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.870914 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.871131 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.871343 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:31.871343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:31.871344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:31.871401 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:31.871665 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:31.871666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:31.871707 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:51:31.871764 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:51:31.871855 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 14:51:31.872614 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:31.863799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.98.60:41361)
0925 14:51:31.863843 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:31.863846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:31.863857 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:31.863940 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:31.863941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:31.863951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:31.863998 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:31.864713 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.865442 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.866457 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:31.866782 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:31.866858 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:31.869079 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:31.869080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:31.869081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:31.869087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:31.870840 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.871229 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.871231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.871231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.871296 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:31.871468 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:31.871469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:31.871469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:31.871599 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:31.871656 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:31.872379 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 14:51:31.872460 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:51:31.874190 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: Bootstrap starting.
I20260925 14:51:31.875815 13831 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:31.876093 13831 log.cc:824] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:31.891230 13831 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: No bootstrap required, opened a new log
I20260925 14:51:31.892917 13831 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.893038 13831 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:31.893059 13831 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059e4d3ba4a845c59403a78e22a36f7b, State: Initialized, Role: FOLLOWER
I20260925 14:51:31.893204 13831 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } }
I20260925 14:51:31.893481 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.893558 13831 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:31.893553 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:51:31.895890 13805 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040966ee543b49fa895dd0dc3efb1b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059e4d3ba4a845c59403a78e22a36f7b" is_pre_election: true
I20260925 14:51:31.896557 13803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:31.897636 13803 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [term 1 FOLLOWER]: Refusing update from remote peer 040966ee543b49fa895dd0dc3efb1b73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:31.897946 13920 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:51:31.898361 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:51:31.898437 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:51:31.895287 13804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "040966ee543b49fa895dd0dc3efb1b73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "059e4d3ba4a845c59403a78e22a36f7b"
I20260925 14:51:31.899497 13804 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 040966ee543b49fa895dd0dc3efb1b73 in term 1.
I20260925 14:51:31.900739 13949 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:31.901473 13949 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:31.901657 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 040966ee543b49fa895dd0dc3efb1b73. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.901727 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:51:31.901767 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.901796 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:51:31.902249 13969 mvcc.cc:205] Tried to move back new op lower bound from 7333264965058023424 to 7333264965048463360. Current Snapshot: MvccSnapshot[applied={T|T < 7333264965058023424}]
I20260925 14:51:31.902954 13920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "040966ee543b49fa895dd0dc3efb1b73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82720c44ce3a4ec38aa26da79be4e951" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41907 } } peers { permanent_uuid: "059e4d3ba4a845c59403a78e22a36f7b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43133 } } peers { permanent_uuid: "040966ee543b49fa895dd0dc3efb1b73" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41361 } } }
I20260925 14:51:31.903029 13920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:31.913911 13949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 040966ee543b49fa895dd0dc3efb1b73: Generated new TSK 0
I20260925 14:51:31.914569 13949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:51:32.852133 13945 catalog_manager.cc:1259] Loaded cluster ID: 2975a6d3ba3241f9b75e6462177a058e
I20260925 14:51:32.852211 13945 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: loading cluster ID for follower catalog manager: success
I20260925 14:51:32.853047 13945 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: acquiring CA information for follower catalog manager: success
I20260925 14:51:32.853366 13945 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82720c44ce3a4ec38aa26da79be4e951: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:51:32.898895 13967 catalog_manager.cc:1259] Loaded cluster ID: 2975a6d3ba3241f9b75e6462177a058e
I20260925 14:51:32.899015 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: loading cluster ID for follower catalog manager: success
I20260925 14:51:32.899825 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: acquiring CA information for follower catalog manager: success
I20260925 14:51:32.900127 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 059e4d3ba4a845c59403a78e22a36f7b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:51:32.976938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13688
I20260925 14:51:32.982187 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13763
I20260925 14:51:32.987005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13838
2026-09-25T14:51:32Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2800 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:33 dist-test-slave-z2s5 krb5kdc[13980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:33 dist-test-slave-z2s5 krb5kdc[13980](info): set up 2 sockets
Sep 25 14:51:33 dist-test-slave-z2s5 krb5kdc[13980](info): commencing operation
krb5kdc: starting...
W20260925 14:51:34.022986 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:34 dist-test-slave-z2s5 krb5kdc[13980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:34.149567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38271
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:34.229588 13993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:34.229802 13993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:34.229825 13993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:34.230955 13993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:34.230997 13993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:34.231011 13993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:34.231024 13993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:34.231036 13993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:34.232406 13993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:34.232592 13993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:34.232760 13993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:34.237432 13999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:34.237440 13998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:34.237556 13993 server_base.cc:1034] running on GCE node
W20260925 14:51:34.237476 14001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:34.238006 13993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:34.238256 13993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:34.239395 13993 hybrid_clock.cc:648] HybridClock initialized: now 1790347894239378 us; error 37 us; skew 500 ppm
Sep 25 14:51:34 dist-test-slave-z2s5 krb5kdc[13980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:34.241923 13993 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:34.242456 13993 webserver.cc:460] Webserver started at http://127.0.98.62:45845/ using document root <none> and password file <none>
I20260925 14:51:34.242681 13993 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:34.242726 13993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:34.242852 13993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:34.243983 13993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "3a88dcf16ac84d24abd4022b78ed6918"
format_stamp: "Formatted at 2026-09-25 14:51:34 on dist-test-slave-z2s5"
server_key: "408f36d28a49d08a37ce084659960453"
server_key_iv: "377ecfba3c38770fb230fd6df4925771"
server_key_version: "encryptionkey@0"
I20260925 14:51:34.244342 13993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a88dcf16ac84d24abd4022b78ed6918"
format_stamp: "Formatted at 2026-09-25 14:51:34 on dist-test-slave-z2s5"
server_key: "408f36d28a49d08a37ce084659960453"
server_key_iv: "377ecfba3c38770fb230fd6df4925771"
server_key_version: "encryptionkey@0"
I20260925 14:51:34.245693 13993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:34.246520 14008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:34.246739 13993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:34.246810 13993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "3a88dcf16ac84d24abd4022b78ed6918"
format_stamp: "Formatted at 2026-09-25 14:51:34 on dist-test-slave-z2s5"
server_key: "408f36d28a49d08a37ce084659960453"
server_key_iv: "377ecfba3c38770fb230fd6df4925771"
server_key_version: "encryptionkey@0"
I20260925 14:51:34.246893 13993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:34.266680 13993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:34.269071 13993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:34.269213 13993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:34.273414 13993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38271
I20260925 14:51:34.273473 14060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38271 every 8 connection(s)
I20260925 14:51:34.273823 13993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:34.274335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 13993
I20260925 14:51:34.274461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:34.274672 392 external_mini_cluster.cc:1428] Setting key 6aa51cf8a063faa01de4226c73bc2e79
I20260925 14:51:34.274782 14061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:34.278117 14061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918: Bootstrap starting.
I20260925 14:51:34.278838 14061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:34.279153 14061 log.cc:824] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:34.280005 14061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918: No bootstrap required, opened a new log
Sep 25 14:51:34 dist-test-slave-z2s5 krb5kdc[13980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:34.282583 14061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a88dcf16ac84d24abd4022b78ed6918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38271 } }
I20260925 14:51:34.282773 14061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:34.282812 14061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a88dcf16ac84d24abd4022b78ed6918, State: Initialized, Role: FOLLOWER
I20260925 14:51:34.282956 14061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a88dcf16ac84d24abd4022b78ed6918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38271 } }
I20260925 14:51:34.283070 14061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:34.283109 14061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:34.283156 14061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:34.283842 14061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a88dcf16ac84d24abd4022b78ed6918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38271 } }
I20260925 14:51:34.283957 14061 leader_election.cc:302] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a88dcf16ac84d24abd4022b78ed6918; no voters:
I20260925 14:51:34.284137 14061 leader_election.cc:288] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:34.284224 14066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:34.284392 14061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:34.284402 14066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [term 1 LEADER]: Becoming Leader. State: Replica: 3a88dcf16ac84d24abd4022b78ed6918, State: Running, Role: LEADER
I20260925 14:51:34.284538 14066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a88dcf16ac84d24abd4022b78ed6918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38271 } }
I20260925 14:51:34.284862 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a88dcf16ac84d24abd4022b78ed6918. Latest consensus state: current_term: 1 leader_uuid: "3a88dcf16ac84d24abd4022b78ed6918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a88dcf16ac84d24abd4022b78ed6918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38271 } } }
I20260925 14:51:34.284874 14068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a88dcf16ac84d24abd4022b78ed6918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a88dcf16ac84d24abd4022b78ed6918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38271 } } }
I20260925 14:51:34.284979 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:34.285228 14068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918 [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:34.285781 14075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:34.286358 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:34.275199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.0.98.62:38271)
0925 14:51:34.275380 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:34.275390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:34.275424 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:51:34.275911 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:34.275917 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:34.275964 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:34.276083 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:34.276887 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:34.277776 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:34.278798 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:34.279299 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:34.279409 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:34.282073 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:34.282079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:34.282080 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:34.282095 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:34.284196 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:34.284629 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:34.284633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:34.284633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:34.284732 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:34.284945 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:34.284946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:34.284946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:34.285299 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:34.285451 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:34.286046 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 14:51:34.286185 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260925 14:51:34.286609 14075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:34.287398 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:51:34.288725 14075 catalog_manager.cc:1347] Generated new cluster ID: 516498635144465d8eef2bf8b9c2d495
I20260925 14:51:34.288766 14075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:34.301184 14075 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:34.301795 14075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:34.308622 14075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a88dcf16ac84d24abd4022b78ed6918: Generated new TSK 0
I20260925 14:51:34.308847 14075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:51:34.341390 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 13993
2026-09-25T14:51:34Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1352 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:34 dist-test-slave-z2s5 krb5kdc[14088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:34 dist-test-slave-z2s5 krb5kdc[14088](info): set up 2 sockets
Sep 25 14:51:34 dist-test-slave-z2s5 krb5kdc[14088](info): commencing operation
krb5kdc: starting...
W20260925 14:51:35.374594 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:35.493274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:35.572307 14101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:35.572518 14101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:35.572540 14101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:35.573683 14101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:35.573724 14101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:35.573737 14101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:35.573776 14101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:35.573805 14101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:35.575263 14101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:35.575479 14101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:35.575695 14101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:35.580322 14109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:35.580327 14106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:35.580356 14107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:35.580494 14101 server_base.cc:1034] running on GCE node
I20260925 14:51:35.580797 14101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:35.581067 14101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:35.582239 14101 hybrid_clock.cc:648] HybridClock initialized: now 1790347895582236 us; error 40 us; skew 500 ppm
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:35.584730 14101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:35.585265 14101 webserver.cc:460] Webserver started at http://127.0.98.62:42977/ using document root <none> and password file <none>
I20260925 14:51:35.585508 14101 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:35.585574 14101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:35.585707 14101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:35.586796 14101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "662f737518dd4091a2a9ec59836b661c"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "24f911d1fc753fd6a2b90d489c3924b1"
server_key_iv: "0e252aef72fec27f84b9699d0531db5a"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.587149 14101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "662f737518dd4091a2a9ec59836b661c"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "24f911d1fc753fd6a2b90d489c3924b1"
server_key_iv: "0e252aef72fec27f84b9699d0531db5a"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.588402 14101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:35.589104 14116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.589264 14101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:35.589362 14101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "662f737518dd4091a2a9ec59836b661c"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "24f911d1fc753fd6a2b90d489c3924b1"
server_key_iv: "0e252aef72fec27f84b9699d0531db5a"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.589442 14101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:35.598754 14101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:35.600941 14101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:35.601109 14101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:35.605193 14101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44123
I20260925 14:51:35.605237 14168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44123 every 8 connection(s)
I20260925 14:51:35.605590 14101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:35.606576 14169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:35.607872 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14101
I20260925 14:51:35.607995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:35.608188 392 external_mini_cluster.cc:1428] Setting key 0ed33bfbd65f15fc88932762b6130e9b
I20260925 14:51:35.609239 14169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap starting.
I20260925 14:51:35.609935 14169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:35.610240 14169 log.cc:824] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:35.610989 14169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: No bootstrap required, opened a new log
I20260925 14:51:35.613139 14169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:35.613304 14169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:35.613337 14169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662f737518dd4091a2a9ec59836b661c, State: Initialized, Role: FOLLOWER
I20260925 14:51:35.613463 14169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:35.613538 14169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:35.613560 14169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:35.613586 14169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:35.614121 14169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:35.614207 14169 leader_election.cc:302] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 662f737518dd4091a2a9ec59836b661c; no voters:
I20260925 14:51:35.614375 14169 leader_election.cc:288] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:35.614439 14174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:35.614584 14174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 LEADER]: Becoming Leader. State: Replica: 662f737518dd4091a2a9ec59836b661c, State: Running, Role: LEADER
I20260925 14:51:35.614663 14169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:35.614756 14174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:35.615577 14176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 662f737518dd4091a2a9ec59836b661c. Latest consensus state: current_term: 1 leader_uuid: "662f737518dd4091a2a9ec59836b661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260925 14:51:35.615661 14176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:35.615726 14175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "662f737518dd4091a2a9ec59836b661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260925 14:51:35.615826 14175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: This master's current role is: LEADER
W20260925 14:51:35.616585 14190 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:51:35.616657 14190 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:51:35.616751 14184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:35.616892 14184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:35.619541 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:35.608710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.0.98.62:44123)
0925 14:51:35.608907 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:35.608917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:35.608948 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:51:35.609477 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:35.609485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:35.609549 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:35.609711 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:35.610331 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.611672 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.612395 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.613092 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.613206 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:35.615967 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:35.615971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:35.615971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:35.615982 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:35.618001 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:35.618350 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:35.618353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:35.618354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:35.618501 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:35.618678 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:35.618679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:35.618679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:35.618965 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:35.619102 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:35.619208 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 14:51:35.619340 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260925 14:51:35.619796 14184 catalog_manager.cc:1347] Generated new cluster ID: 8a5256125f494d1aa3a70710d0d44155
I20260925 14:51:35.619845 14184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:35.636502 14184 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:35.637109 14184 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:35.639715 14184 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Generated new TSK 0
I20260925 14:51:35.639977 14184 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:51:35.674286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:35.776621 14195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:35.776866 14195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:35.776908 14195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:35.778050 14195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:35.778118 14195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:35.778170 14195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:35.779735 14195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:35.779995 14195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:35.780226 14195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:35.785245 14200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:35.785226 14201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:35.785226 14203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:35.785686 14195 server_base.cc:1034] running on GCE node
I20260925 14:51:35.785873 14195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:35.786085 14195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:35.787230 14195 hybrid_clock.cc:648] HybridClock initialized: now 1790347895787220 us; error 32 us; skew 500 ppm
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:35.789937 14195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:35.790460 14195 webserver.cc:460] Webserver started at http://127.0.98.1:46323/ using document root <none> and password file <none>
I20260925 14:51:35.790685 14195 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:35.790752 14195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:35.790905 14195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:35.791977 14195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "4110f98847ef4f73b05035c45d306d29"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "602ac2567d0573c43404b7e76b9c1596"
server_key_iv: "d30a9956f9e733d82d9e95d753e4eede"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.792395 14195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4110f98847ef4f73b05035c45d306d29"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "602ac2567d0573c43404b7e76b9c1596"
server_key_iv: "d30a9956f9e733d82d9e95d753e4eede"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.793761 14195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:35.794521 14210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.794710 14195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:35.794806 14195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "4110f98847ef4f73b05035c45d306d29"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "602ac2567d0573c43404b7e76b9c1596"
server_key_iv: "d30a9956f9e733d82d9e95d753e4eede"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.794893 14195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:35.807757 14195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:35.810508 14195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:35.810678 14195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:35.810930 14195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:35.811246 14195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:35.811301 14195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.811336 14195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:35.811384 14195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.817448 14195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41737
I20260925 14:51:35.817494 14323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41737 every 8 connection(s)
I20260925 14:51:35.817807 14195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:35.820062 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14195
I20260925 14:51:35.820196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:35.820416 392 external_mini_cluster.cc:1428] Setting key 4a00e87c572f59ee1e2e9dcd41b63fbc
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:35.828039 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:35.818597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44937 (local address 127.0.98.62:44123)
0925 14:51:35.818714 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:35.818716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:35.818910 (+ 194us) server_negotiation.cc:407] Connection header received
0925 14:51:35.819443 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:35.819445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:35.819457 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:35.819522 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:35.820586 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.821106 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.822120 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.822464 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.822548 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:35.824996 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:35.824998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:35.824998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:35.825007 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:35.826488 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:35.827019 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:35.827020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:35.827020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:35.827069 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:35.827366 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:35.827367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:35.827367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:35.827465 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:35.827524 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:35.827886 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 14:51:35.827972 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:51:35.828351 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:35.818564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:44937)
0925 14:51:35.818777 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:35.818825 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:35.819337 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:35.819600 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:35.819604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:35.820248 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:35.820475 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:35.820480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.821214 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.821215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:35.822020 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:35.822022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.822565 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.822566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:35.822687 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:35.823253 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:35.823262 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:35.824902 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:35.826570 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:35.826578 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:35.826581 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:35.826939 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:35.827139 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:35.827140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:35.827140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:35.827297 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:35.827542 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:35.827549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:35.827733 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 14:51:35.827875 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 14:51:35.828068 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
I20260925 14:51:35.828954 14324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:51:35.829094 14324 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:35.829274 14324 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260925 14:51:35.829814 14133 ts_manager.cc:194] Registered new tserver with Master: 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737)
I20260925 14:51:35.830514 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:51:35.847635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:35.927528 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:35.927728 14329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:35.927752 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:35.928829 14329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:35.928866 14329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:35.928891 14329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:35.930250 14329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:35.930509 14329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:35.930699 14329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:51:35.935711 14329 server_base.cc:1034] running on GCE node
W20260925 14:51:35.935662 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:35.935662 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:35.935681 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:35.936123 14329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:35.936337 14329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:35.937536 14329 hybrid_clock.cc:648] HybridClock initialized: now 1790347895937523 us; error 26 us; skew 500 ppm
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:35.940217 14329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:35.940606 14329 webserver.cc:460] Webserver started at http://127.0.98.2:44687/ using document root <none> and password file <none>
I20260925 14:51:35.940814 14329 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:35.940853 14329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:35.940939 14329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:35.941931 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "669d40be33aa4a11a4188d2c95f39de2"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "f6449ced5c79ee9ff60dd995ddbfe3f2"
server_key_iv: "7e06bf4879ceb6d2e3f25ce5b183b154"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.942237 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "669d40be33aa4a11a4188d2c95f39de2"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "f6449ced5c79ee9ff60dd995ddbfe3f2"
server_key_iv: "7e06bf4879ceb6d2e3f25ce5b183b154"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.943519 14329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:51:35.944177 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.944329 14329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:35.944409 14329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "669d40be33aa4a11a4188d2c95f39de2"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "f6449ced5c79ee9ff60dd995ddbfe3f2"
server_key_iv: "7e06bf4879ceb6d2e3f25ce5b183b154"
server_key_version: "encryptionkey@0"
I20260925 14:51:35.944463 14329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:35.958724 14329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:35.961014 14329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:35.961128 14329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:35.961308 14329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:35.961625 14329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:35.961659 14329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.961704 14329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:35.961731 14329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:35.967315 14329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42211
I20260925 14:51:35.967401 14457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42211 every 8 connection(s)
I20260925 14:51:35.967701 14329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:35.972596 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14329
I20260925 14:51:35.972673 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:35.972863 392 external_mini_cluster.cc:1428] Setting key dc6eb6c77653c4b5dc27f3bff795c9d8
Sep 25 14:51:35 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:35.978068 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:35.968469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56609 (local address 127.0.98.62:44123)
0925 14:51:35.968550 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:35.968553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:35.968722 (+ 169us) server_negotiation.cc:407] Connection header received
0925 14:51:35.969195 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:35.969197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:35.969207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:35.969260 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:35.970151 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.970722 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.971692 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.971872 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.971921 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:35.974407 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:35.974409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:35.974409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:35.974417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:35.976159 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:35.976721 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:35.976722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:35.976722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:35.976782 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:35.977236 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:35.977237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:35.977238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:35.977408 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:35.977482 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:35.977896 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 14:51:35.977987 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 14:51:35.978361 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:35.968420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:56609)
0925 14:51:35.968593 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:35.968630 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:35.969111 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:35.969314 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:35.969318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:35.969886 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:35.970077 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:35.970081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.970800 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.970800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:35.971616 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:35.971618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:35.971942 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:35.971943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:35.972049 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:35.972599 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:35.972608 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:35.974298 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:35.976250 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:35.976258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:35.976261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:35.976615 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:35.976860 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:35.976861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:35.976861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:35.977049 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:35.977491 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:35.977499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:35.977706 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 14:51:35.977838 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 14:51:35.978033 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 14:51:35.978986 14458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260925 14:51:35.979111 14458 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:35.979310 14458 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260925 14:51:35.979768 14133 ts_manager.cc:194] Registered new tserver with Master: 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211)
I20260925 14:51:35.980206 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56609
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:51:35.999150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:36.086946 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:36.087163 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:36.087189 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:36.088451 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:36.088496 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:36.088526 14463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:36.090219 14463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:36.090502 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:36.090745 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:36.096230 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.096240 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:36.096421 14463 server_base.cc:1034] running on GCE node
W20260925 14:51:36.096251 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:36.096746 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:36.097014 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:36.098193 14463 hybrid_clock.cc:648] HybridClock initialized: now 1790347896098185 us; error 36 us; skew 500 ppm
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:36.100845 14463 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:36.101341 14463 webserver.cc:460] Webserver started at http://127.0.98.3:34981/ using document root <none> and password file <none>
I20260925 14:51:36.101583 14463 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:36.101666 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:36.101830 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:36.103085 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "6c57b36563604791bd6215a77faa3ddf"
format_stamp: "Formatted at 2026-09-25 14:51:36 on dist-test-slave-z2s5"
server_key: "69b58e948a554c7813fa05ca9a970dc4"
server_key_iv: "2a9105c3bf03d1d8f6deff26248952b3"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.103515 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6c57b36563604791bd6215a77faa3ddf"
format_stamp: "Formatted at 2026-09-25 14:51:36 on dist-test-slave-z2s5"
server_key: "69b58e948a554c7813fa05ca9a970dc4"
server_key_iv: "2a9105c3bf03d1d8f6deff26248952b3"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.104980 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:36.105804 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.105981 14463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.106107 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "6c57b36563604791bd6215a77faa3ddf"
format_stamp: "Formatted at 2026-09-25 14:51:36 on dist-test-slave-z2s5"
server_key: "69b58e948a554c7813fa05ca9a970dc4"
server_key_iv: "2a9105c3bf03d1d8f6deff26248952b3"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.106187 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:36.117448 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:36.119772 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:36.119892 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:36.120070 14463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:36.120390 14463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:36.120448 14463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.120492 14463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:36.120518 14463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.126375 14463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46231
I20260925 14:51:36.126475 14591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46231 every 8 connection(s)
I20260925 14:51:36.126842 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:36.134033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14463
I20260925 14:51:36.134151 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:36.134330 392 external_mini_cluster.cc:1428] Setting key 439fa4bea07f665239d02fe0b0bd27ee
I20260925 14:51:36.136480 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.127623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50483 (local address 127.0.98.62:44123)
0925 14:51:36.127693 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.127695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.127914 (+ 219us) server_negotiation.cc:407] Connection header received
0925 14:51:36.128387 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.128388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.128398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.128464 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.129417 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.129938 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.130962 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.131154 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.131210 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.133366 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.133367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.133368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.133374 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.134965 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.135587 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.135588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.135588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.135647 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.135857 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.135857 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.135858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.135972 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.136028 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.136329 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.136401 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:51:36.136757 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.127609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:50483)
0925 14:51:36.127785 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.127828 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.128306 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.128519 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.128522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.129088 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:36.129309 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.129324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.130022 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.130023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.130885 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.130888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.131228 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.131229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.131326 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.131859 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:36.131868 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:36.133282 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:36.135143 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.135150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.135153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.135505 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.135695 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.135696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.135696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.135800 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.136057 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:36.136064 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:36.136199 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.136317 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.136465 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260925 14:51:36.137276 14592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260925 14:51:36.137403 14592 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:36.137604 14592 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260925 14:51:36.138022 14133 ts_manager.cc:194] Registered new tserver with Master: 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
I20260925 14:51:36.138440 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50483
I20260925 14:51:36.146801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:51:36.154516 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.148330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37888 (local address 127.0.98.62:44123)
0925 14:51:36.148395 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.148397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.148455 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:51:36.148504 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.148505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.148514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.148575 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.149099 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.149631 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.150320 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.150616 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.150678 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.151813 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.151815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.151815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.151822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.153399 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.153754 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.153756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.153756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.153813 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.153986 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.153987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.153987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.154138 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.154216 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.154301 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.154363 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:51:36.156160 14133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:36.158061 14133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:36.164881 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.160603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41737 (local address 127.0.0.1:45500)
0925 14:51:36.160966 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.161020 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.161095 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.161397 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.161398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.161400 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.161452 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.161453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.162418 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.162419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.163706 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.163709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.164540 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.164620 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.164634 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.164733 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.164786 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":59,"threads_started":1}
I20260925 14:51:36.165089 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.160732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45500 (local address 127.0.98.1:41737)
0925 14:51:36.160922 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.160928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.161176 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:51:36.161240 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.161241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.161254 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.161345 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.161601 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.162288 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.163784 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.164430 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164527 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.164592 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.164925 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.164996 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":75,"threads_started":1}
I20260925 14:51:36.165144 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.160704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42211 (local address 127.0.0.1:60754)
0925 14:51:36.160994 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.161028 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.161094 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.161349 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.161352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.161354 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.161429 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.161431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.162824 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.162825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.163844 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.163846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.164729 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.164782 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.164785 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.164837 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.164871 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260925 14:51:36.165323 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.160948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60754 (local address 127.0.98.2:42211)
0925 14:51:36.161198 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.161203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.161212 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:51:36.161252 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.161254 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.161269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.161344 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.161531 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.162715 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.163938 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.164645 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164723 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.164773 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.164852 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.164895 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":85,"threads_started":1}
I20260925 14:51:36.165323 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.160917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.0.98.3:46231)
0925 14:51:36.161098 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.161103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.161555 (+ 452us) server_negotiation.cc:407] Connection header received
0925 14:51:36.161600 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.161602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.161617 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.161681 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.161867 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.162921 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164142 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.164827 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164889 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.164937 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.165191 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.165235 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260925 14:51:36.165501 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.160822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46231 (local address 127.0.0.1:60090)
0925 14:51:36.161436 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.161446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.161493 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.161755 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.161756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.161757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.161792 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.161794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.163028 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.163029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.164053 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.164055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.164972 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.164973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.165058 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.165062 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.165109 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.165151 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":82,"threads_started":1}
I20260925 14:51:36.165954 14258 tablet_service.cc:1467] Processing CreateTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 (DEFAULT_TABLE table=test-table [id=d80e1302320247feb29bf4affd3f5aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:36.166006 14392 tablet_service.cc:1467] Processing CreateTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 (DEFAULT_TABLE table=test-table [id=d80e1302320247feb29bf4affd3f5aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:36.166312 14392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7208e3a51eeb420cb063e5380fd1ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:36.166312 14258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7208e3a51eeb420cb063e5380fd1ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:36.168879 14609 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Bootstrap starting.
I20260925 14:51:36.169272 14526 tablet_service.cc:1467] Processing CreateTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 (DEFAULT_TABLE table=test-table [id=d80e1302320247feb29bf4affd3f5aa1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:36.169480 14609 tablet_bootstrap.cc:654] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:36.169586 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7208e3a51eeb420cb063e5380fd1ca84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:36.169986 14609 log.cc:824] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.170684 14610 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Bootstrap starting.
I20260925 14:51:36.170768 14609 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: No bootstrap required, opened a new log
I20260925 14:51:36.170871 14609 ts_tablet_manager.cc:1397] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:36.171321 14610 tablet_bootstrap.cc:654] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:36.171877 14610 log.cc:824] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.172509 14613 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Bootstrap starting.
I20260925 14:51:36.172720 14610 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: No bootstrap required, opened a new log
I20260925 14:51:36.172765 14610 ts_tablet_manager.cc:1397] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:36.173115 14613 tablet_bootstrap.cc:654] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:36.173214 14609 raft_consensus.cc:348] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.173400 14609 raft_consensus.cc:374] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:36.173441 14609 raft_consensus.cc:729] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4110f98847ef4f73b05035c45d306d29, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.173571 14609 consensus_queue.cc:260] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.173630 14613 log.cc:824] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.173854 14609 ts_tablet_manager.cc:1428] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:36.173878 14324 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
I20260925 14:51:36.174362 14613 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: No bootstrap required, opened a new log
I20260925 14:51:36.174446 14613 ts_tablet_manager.cc:1397] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:36.174479 14610 raft_consensus.cc:348] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.174654 14610 raft_consensus.cc:374] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:36.174698 14610 raft_consensus.cc:729] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669d40be33aa4a11a4188d2c95f39de2, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.174808 14610 consensus_queue.cc:260] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.175073 14610 ts_tablet_manager.cc:1428] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:51:36.175088 14458 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
I20260925 14:51:36.176014 14613 raft_consensus.cc:348] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.176136 14613 raft_consensus.cc:374] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:36.176156 14613 raft_consensus.cc:729] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57b36563604791bd6215a77faa3ddf, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.176286 14613 consensus_queue.cc:260] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.176605 14613 ts_tablet_manager.cc:1428] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:51:36.176661 14592 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
W20260925 14:51:36.218286 14459 tablet.cc:2369] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:36.284392 14616 raft_consensus.cc:482] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:36.284535 14616 raft_consensus.cc:504] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.284940 14616 leader_election.cc:288] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
I20260925 14:51:36.289106 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.285428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43635 (local address 127.0.98.3:46231)
0925 14:51:36.285475 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.285477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.285625 (+ 148us) server_negotiation.cc:407] Connection header received
0925 14:51:36.285653 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.285654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.285667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.285754 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.285890 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.286762 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.288064 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.288702 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.288804 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.288815 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.288979 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.289033 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:51:36.289106 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.285273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35777 (local address 127.0.98.1:41737)
0925 14:51:36.285336 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.285338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.285375 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:51:36.285462 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.285464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.285478 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.285561 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.285727 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.286642 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.288064 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.288720 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.288804 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.288815 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.288979 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.289026 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:51:36.289116 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.285200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41737 (local address 127.0.98.2:35777)
0925 14:51:36.285306 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.285321 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.285393 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.285570 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.285571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.285573 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.285657 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.285658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.286725 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.286726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.287945 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.287948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.288812 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.288813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.288896 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.288898 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.288953 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.289035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":10}
I20260925 14:51:36.289125 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.285344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46231 (local address 127.0.98.2:43635)
0925 14:51:36.285527 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.285537 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.285585 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.285772 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.285773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.285779 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.285825 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.285826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.286835 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.286836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.287945 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.287948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.288812 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.288813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.288892 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.288896 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.288953 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.289000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":10,"thread_start_us":122,"threads_started":1}
I20260925 14:51:36.289491 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7208e3a51eeb420cb063e5380fd1ca84" candidate_uuid: "669d40be33aa4a11a4188d2c95f39de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57b36563604791bd6215a77faa3ddf" is_pre_election: true
I20260925 14:51:36.289649 14546 raft_consensus.cc:2454] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669d40be33aa4a11a4188d2c95f39de2 in term 0.
I20260925 14:51:36.289606 14278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7208e3a51eeb420cb063e5380fd1ca84" candidate_uuid: "669d40be33aa4a11a4188d2c95f39de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4110f98847ef4f73b05035c45d306d29" is_pre_election: true
I20260925 14:51:36.289772 14278 raft_consensus.cc:2454] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669d40be33aa4a11a4188d2c95f39de2 in term 0.
I20260925 14:51:36.289865 14348 leader_election.cc:302] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669d40be33aa4a11a4188d2c95f39de2, 6c57b36563604791bd6215a77faa3ddf; no voters:
I20260925 14:51:36.290027 14616 raft_consensus.cc:2781] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:36.290089 14616 raft_consensus.cc:482] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:36.290117 14616 raft_consensus.cc:3037] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:36.291010 14616 raft_consensus.cc:504] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.291150 14616 leader_election.cc:288] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [CANDIDATE]: Term 1 election: Requested vote from peers 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
I20260925 14:51:36.291299 14278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7208e3a51eeb420cb063e5380fd1ca84" candidate_uuid: "669d40be33aa4a11a4188d2c95f39de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4110f98847ef4f73b05035c45d306d29"
I20260925 14:51:36.291344 14546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7208e3a51eeb420cb063e5380fd1ca84" candidate_uuid: "669d40be33aa4a11a4188d2c95f39de2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57b36563604791bd6215a77faa3ddf"
I20260925 14:51:36.291369 14278 raft_consensus.cc:3037] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:36.291450 14546 raft_consensus.cc:3037] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:36.292153 14278 raft_consensus.cc:2454] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669d40be33aa4a11a4188d2c95f39de2 in term 1.
I20260925 14:51:36.292277 14546 raft_consensus.cc:2454] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669d40be33aa4a11a4188d2c95f39de2 in term 1.
I20260925 14:51:36.292441 14348 leader_election.cc:302] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29, 669d40be33aa4a11a4188d2c95f39de2; no voters:
I20260925 14:51:36.292555 14616 raft_consensus.cc:2781] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:36.292733 14616 raft_consensus.cc:686] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 LEADER]: Becoming Leader. State: Replica: 669d40be33aa4a11a4188d2c95f39de2, State: Running, Role: LEADER
I20260925 14:51:36.292842 14616 consensus_queue.cc:237] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.293715 14133 catalog_manager.cc:5495] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "669d40be33aa4a11a4188d2c95f39de2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } health_report { overall_health: HEALTHY } } }
I20260925 14:51:36.309417 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.306985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60768 (local address 127.0.98.2:42211)
0925 14:51:36.307081 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.307083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.307097 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:51:36.307128 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.307128 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.307147 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.307226 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:36.307359 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.307824 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.308600 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.308800 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.308880 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.309059 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:36.309160 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:36.309248 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.309291 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.309312 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 14:51:36.311465 14546 raft_consensus.cc:1261] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Refusing update from remote peer 669d40be33aa4a11a4188d2c95f39de2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:36.311512 14278 raft_consensus.cc:1261] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Refusing update from remote peer 669d40be33aa4a11a4188d2c95f39de2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:36.311887 14619 consensus_queue.cc:1035] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:36.312037 14616 consensus_queue.cc:1035] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:36.314466 14622 mvcc.cc:205] Tried to move back new op lower bound from 7333264983289180160 to 7333264983216136192. Current Snapshot: MvccSnapshot[applied={T|T < 7333264983289180160}]
I20260925 14:51:36.314714 14623 mvcc.cc:205] Tried to move back new op lower bound from 7333264983289180160 to 7333264983216136192. Current Snapshot: MvccSnapshot[applied={T|T < 7333264983289180160}]
I20260925 14:51:36.316232 14624 mvcc.cc:205] Tried to move back new op lower bound from 7333264983289180160 to 7333264983216136192. Current Snapshot: MvccSnapshot[applied={T|T < 7333264983289180160 or (T in {7333264983289180160})}]
I20260925 14:51:36.317663 14132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:36.317760 14132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:36.318624 14132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Sending DeleteTablet for 3 replicas of tablet 7208e3a51eeb420cb063e5380fd1ca84
I20260925 14:51:36.318960 14258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45500
I20260925 14:51:36.318959 14392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60754
I20260925 14:51:36.319101 14526 tablet_service.cc:1514] Processing DeleteTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60090
I20260925 14:51:36.319221 14633 tablet_replica.cc:330] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: stopping tablet replica
I20260925 14:51:36.319258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14195
I20260925 14:51:36.319335 14633 raft_consensus.cc:2229] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:36.319438 14634 tablet_replica.cc:330] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: stopping tablet replica
I20260925 14:51:36.319458 14633 raft_consensus.cc:2258] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:36.319619 14632 tablet_replica.cc:330] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: stopping tablet replica
I20260925 14:51:36.319710 14632 raft_consensus.cc:2229] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:36.319800 14632 raft_consensus.cc:2258] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:36.319964 14634 raft_consensus.cc:2229] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:36.320111 14634 raft_consensus.cc:2258] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:36.320634 14634 ts_tablet_manager.cc:1905] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:36.321895 14632 ts_tablet_manager.cc:1905] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:36.322477 14634 ts_tablet_manager.cc:1918] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:36.322618 14634 log.cc:1196] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/7208e3a51eeb420cb063e5380fd1ca84
I20260925 14:51:36.322955 14634 ts_tablet_manager.cc:1939] T 7208e3a51eeb420cb063e5380fd1ca84 P 6c57b36563604791bd6215a77faa3ddf: Deleting consensus metadata
I20260925 14:51:36.323421 14120 catalog_manager.cc:4841] TS 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231): tablet 7208e3a51eeb420cb063e5380fd1ca84 (table test-table [id=d80e1302320247feb29bf4affd3f5aa1]) successfully deleted
I20260925 14:51:36.324190 14632 ts_tablet_manager.cc:1918] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:36.324273 14632 log.cc:1196] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/7208e3a51eeb420cb063e5380fd1ca84
I20260925 14:51:36.324561 14632 ts_tablet_manager.cc:1939] T 7208e3a51eeb420cb063e5380fd1ca84 P 669d40be33aa4a11a4188d2c95f39de2: Deleting consensus metadata
I20260925 14:51:36.325655 14119 catalog_manager.cc:4841] TS 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211): tablet 7208e3a51eeb420cb063e5380fd1ca84 (table test-table [id=d80e1302320247feb29bf4affd3f5aa1]) successfully deleted
W20260925 14:51:36.327266 14120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41737 (error 108)
I20260925 14:51:36.327529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14329
I20260925 14:51:36.329051 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.328902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41737 (local address 127.0.0.1:45504)
0925 14:51:36.328973 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.329025 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 14:51:36.329175 14120 catalog_manager.cc:4584] TS 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7208e3a51eeb420cb063e5380fd1ca84: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41737: connect: Connection refused (error 111)
I20260925 14:51:36.333631 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14463
I20260925 14:51:36.338426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14101
I20260925 14:51:36.343662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--webserver_interface=127.0.98.62
--webserver_port=42977
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:36.427727 14636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:36.428001 14636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:36.428045 14636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:36.429234 14636 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:36.429311 14636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:36.429337 14636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:36.429365 14636 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:36.429399 14636 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:36.430958 14636 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14636
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:36.431259 14636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:36.431500 14636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:36.440270 14643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.440256 14642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.440258 14645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:36.440582 14636 server_base.cc:1034] running on GCE node
I20260925 14:51:36.440802 14636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:36.441082 14636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:36.442220 14636 hybrid_clock.cc:648] HybridClock initialized: now 1790347896442192 us; error 49 us; skew 500 ppm
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:36.444844 14636 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:36.445441 14636 webserver.cc:460] Webserver started at http://127.0.98.62:42977/ using document root <none> and password file <none>
I20260925 14:51:36.445698 14636 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:36.445758 14636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:36.447270 14636 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.447979 14652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.448196 14636 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.448283 14636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "662f737518dd4091a2a9ec59836b661c"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "24f911d1fc753fd6a2b90d489c3924b1"
server_key_iv: "0e252aef72fec27f84b9699d0531db5a"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.448684 14636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:36.461920 14636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:36.464001 14636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:36.464118 14636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:36.468082 14636 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44123
I20260925 14:51:36.468200 14704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44123 every 8 connection(s)
I20260925 14:51:36.468598 14636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:36.470111 14705 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:51:36.470912 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap starting.
I20260925 14:51:36.478711 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14636
I20260925 14:51:36.479069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41737
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44123
--webserver_port=46323
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:36.479475 14705 log.cc:824] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.481986 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:51:36.482297 14705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap complete.
I20260925 14:51:36.486521 14705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:36.486955 14705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662f737518dd4091a2a9ec59836b661c, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.487147 14705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:36.487237 14705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:36.487290 14705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:36.487349 14705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:51:36.489049 14705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:36.489184 14705 leader_election.cc:302] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 662f737518dd4091a2a9ec59836b661c; no voters:
I20260925 14:51:36.489377 14705 leader_election.cc:288] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:51:36.489466 14709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:51:36.489691 14705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:36.489698 14709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 LEADER]: Becoming Leader. State: Replica: 662f737518dd4091a2a9ec59836b661c, State: Running, Role: LEADER
I20260925 14:51:36.489921 14709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:36.490270 14710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "662f737518dd4091a2a9ec59836b661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260925 14:51:36.490351 14710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:36.490561 14711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 662f737518dd4091a2a9ec59836b661c. Latest consensus state: current_term: 2 leader_uuid: "662f737518dd4091a2a9ec59836b661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260925 14:51:36.490630 14711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:36.490875 14719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:36.491698 14719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:36.491936 14719 catalog_manager.cc:1259] Loaded cluster ID: 8a5256125f494d1aa3a70710d0d44155
I20260925 14:51:36.492048 14719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:51:36.492285 14719 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 14:51:36.495214 14719 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 14:51:36.499931 14719 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:36.500175 14719 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Loaded TSK: 0
I20260925 14:51:36.500407 14719 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:51:36.577713 14708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:36.577944 14708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:36.577983 14708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:36.579250 14708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:36.579321 14708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:36.579367 14708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:36.580829 14708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46323
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:36.581100 14708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:36.581342 14708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:36.586503 14733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.586539 14734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.586707 14736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:36.586984 14708 server_base.cc:1034] running on GCE node
I20260925 14:51:36.587231 14708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:36.587487 14708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:36.588653 14708 hybrid_clock.cc:648] HybridClock initialized: now 1790347896588639 us; error 31 us; skew 500 ppm
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:36.591650 14708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:36.592250 14708 webserver.cc:460] Webserver started at http://127.0.98.1:46323/ using document root <none> and password file <none>
I20260925 14:51:36.592537 14708 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:36.592638 14708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:36.594465 14708 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:36.595331 14743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.595520 14708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.595642 14708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "4110f98847ef4f73b05035c45d306d29"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "602ac2567d0573c43404b7e76b9c1596"
server_key_iv: "d30a9956f9e733d82d9e95d753e4eede"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.596019 14708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:36.623464 14708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:36.625993 14708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:36.626150 14708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:36.626492 14708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:36.626997 14750 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:51:36.628832 14708 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:51:36.628906 14708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:51:36.628955 14708 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:51:36.629663 14708 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:51:36.629724 14708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:36.629766 14750 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Bootstrap starting.
I20260925 14:51:36.637764 14708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41737
I20260925 14:51:36.637856 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41737 every 8 connection(s)
I20260925 14:51:36.638330 14708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:36.640740 14750 log.cc:824] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.643149 14750 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:51:36.643457 14750 tablet_bootstrap.cc:492] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Bootstrap complete.
I20260925 14:51:36.644562 14750 ts_tablet_manager.cc:1397] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260925 14:51:36.645361 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14708
I20260925 14:51:36.645651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42211
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44123
--webserver_port=44687
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:36.646035 14750 raft_consensus.cc:348] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.646485 14750 raft_consensus.cc:729] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4110f98847ef4f73b05035c45d306d29, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.646678 14750 consensus_queue.cc:260] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } }
I20260925 14:51:36.647214 14750 ts_tablet_manager.cc:1428] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:51:36.650801 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.639311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34849 (local address 127.0.98.62:44123)
0925 14:51:36.639489 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.639504 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.639533 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:51:36.640376 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.640382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.640425 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.640555 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.641396 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.642263 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.643806 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.644201 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.644283 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.646908 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.646912 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.646913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.646927 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.648890 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.649444 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.649447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.649447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.649555 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.649862 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.649863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.649863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.650088 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.650155 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.650552 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.650663 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260925 14:51:36.650951 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.639165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:34849)
0925 14:51:36.639328 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.639370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.640146 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.640636 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.640640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.641015 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:36.641273 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.641280 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.642386 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.642414 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.643724 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.643727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.644291 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.644292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.644449 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.645066 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:36.645075 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:36.646755 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:36.648988 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.648995 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.648998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.649340 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.649634 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.649635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.649635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.649786 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.650163 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:36.650170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:36.650350 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.650499 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.650699 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260925 14:51:36.651711 14858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260925 14:51:36.651814 14858 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:36.652022 14858 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260925 14:51:36.652750 14669 ts_manager.cc:194] Registered new tserver with Master: 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737)
I20260925 14:51:36.654098 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34849
I20260925 14:51:36.655234 14858 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:51:36.662582 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.653537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41737 (local address 127.0.0.1:45514)
0925 14:51:36.653828 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.653859 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.653969 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.654262 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.654268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.654685 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:36.654740 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.654743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.655710 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.655712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.656417 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.656420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.656997 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.657001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.657089 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.657677 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:36.657682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:36.659404 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:36.661286 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.661294 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.661296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.661511 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.661753 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.661753 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.661754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.661825 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.662134 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:36.662140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:36.662197 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.662309 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.662486 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260925 14:51:36.662582 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.653619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.98.1:41737)
0925 14:51:36.654093 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.654097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.654105 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:36.654135 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.654137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.654157 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.654258 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.654845 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.655589 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.656539 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.656874 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.656981 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.659502 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.659511 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.659511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.659519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.661172 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.661613 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.661616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.661617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.661675 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.661904 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.661905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.661906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.662055 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.662142 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.662349 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.662478 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":84,"threads_started":1}
I20260925 14:51:36.662981 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 7208e3a51eeb420cb063e5380fd1ca84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45514
I20260925 14:51:36.663515 14867 tablet_replica.cc:330] stopping tablet replica
I20260925 14:51:36.663631 14867 raft_consensus.cc:2229] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:36.663702 14867 raft_consensus.cc:2258] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:36.664016 14867 ts_tablet_manager.cc:1905] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:36.666436 14867 ts_tablet_manager.cc:1918] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:36.666513 14867 log.cc:1196] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/7208e3a51eeb420cb063e5380fd1ca84
I20260925 14:51:36.666759 14867 ts_tablet_manager.cc:1939] T 7208e3a51eeb420cb063e5380fd1ca84 P 4110f98847ef4f73b05035c45d306d29: Deleting consensus metadata
I20260925 14:51:36.667310 14656 catalog_manager.cc:4841] TS 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737): tablet 7208e3a51eeb420cb063e5380fd1ca84 (table test-table [id=d80e1302320247feb29bf4affd3f5aa1]) successfully deleted
W20260925 14:51:36.731103 14864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:36.731303 14864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:36.731328 14864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:36.732501 14864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:36.732550 14864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:36.732620 14864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:36.734063 14864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44687
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:36.734308 14864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:36.734560 14864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:36.739452 14873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.739476 14875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.739452 14872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:36.739938 14864 server_base.cc:1034] running on GCE node
I20260925 14:51:36.740135 14864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:36.740365 14864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:36.741503 14864 hybrid_clock.cc:648] HybridClock initialized: now 1790347896741481 us; error 36 us; skew 500 ppm
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:36.743889 14864 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:36.744309 14864 webserver.cc:460] Webserver started at http://127.0.98.2:44687/ using document root <none> and password file <none>
I20260925 14:51:36.744535 14864 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:36.744590 14864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:36.745899 14864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.746506 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.746712 14864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:36.746803 14864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "669d40be33aa4a11a4188d2c95f39de2"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "f6449ced5c79ee9ff60dd995ddbfe3f2"
server_key_iv: "7e06bf4879ceb6d2e3f25ce5b183b154"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.747146 14864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:36.759825 14864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:36.761735 14864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:36.761893 14864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:36.762135 14864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:36.762495 14864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:36.762539 14864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.762562 14864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:36.762609 14864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.768261 14864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42211
I20260925 14:51:36.768303 14995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42211 every 8 connection(s)
I20260925 14:51:36.768643 14864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:36.771667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14864
I20260925 14:51:36.771960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46231
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44123
--webserver_port=34981
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:36.779484 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.769467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39849 (local address 127.0.98.62:44123)
0925 14:51:36.769515 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.769517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.769784 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:51:36.770370 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.770371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.770417 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.770476 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.771554 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.772271 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.773721 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.773933 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.773995 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.776590 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.776592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.776592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.776601 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.778028 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.778486 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.778488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.778488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.778557 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.778814 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.778815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.778816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.778933 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.779014 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.779294 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.779394 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:51:36.779664 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.769466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:39849)
0925 14:51:36.769651 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.769689 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.770244 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.770581 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.770584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.771215 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:36.771419 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.771424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.772378 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.772379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.773616 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.773619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.774033 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.774035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.774191 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.774768 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:36.774779 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:36.776479 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:36.778132 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.778143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.778145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.778381 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.778627 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.778628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.778629 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.778744 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.779005 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:36.779011 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:36.779162 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.779275 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.779432 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":91,"threads_started":1}
I20260925 14:51:36.780175 14996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260925 14:51:36.780297 14996 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:36.780536 14996 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260925 14:51:36.780964 14669 ts_manager.cc:194] Registered new tserver with Master: 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211)
I20260925 14:51:36.781579 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39849
W20260925 14:51:36.861151 14999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:36.861385 14999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:36.861411 14999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:36.862574 14999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:36.862610 14999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:36.862636 14999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:36.864032 14999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34981
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:36.864233 14999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:36.864403 14999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:36.869673 15004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.869668 15005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:36.869668 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:36.869712 14999 server_base.cc:1034] running on GCE node
I20260925 14:51:36.870136 14999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:36.870417 14999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:36.871577 14999 hybrid_clock.cc:648] HybridClock initialized: now 1790347896871559 us; error 32 us; skew 500 ppm
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:36.873991 14999 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:36.874539 14999 webserver.cc:460] Webserver started at http://127.0.98.3:34981/ using document root <none> and password file <none>
I20260925 14:51:36.874778 14999 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:36.874851 14999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:36.876147 14999 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.876751 15014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.876947 14999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.877053 14999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "6c57b36563604791bd6215a77faa3ddf"
format_stamp: "Formatted at 2026-09-25 14:51:36 on dist-test-slave-z2s5"
server_key: "69b58e948a554c7813fa05ca9a970dc4"
server_key_iv: "2a9105c3bf03d1d8f6deff26248952b3"
server_key_version: "encryptionkey@0"
I20260925 14:51:36.877393 14999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:36.913259 14999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:36.915436 14999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:36.915609 14999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:36.915858 14999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:36.916189 14999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:36.916244 14999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.916286 14999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:36.916314 14999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:36.921892 14999 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46231
I20260925 14:51:36.921938 15127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46231 every 8 connection(s)
I20260925 14:51:36.922267 14999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:36.927979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 14999
Sep 25 14:51:36 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:36.932847 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.923083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41649 (local address 127.0.98.62:44123)
0925 14:51:36.923170 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.923172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.923370 (+ 198us) server_negotiation.cc:407] Connection header received
0925 14:51:36.923951 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.923953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.923967 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.924013 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.925063 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.925633 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.926793 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.927031 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.927126 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.929715 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.929717 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.929717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.929725 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.931259 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.931704 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.931706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.931706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.931770 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.932063 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.932064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.932064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.932184 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.932270 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.932651 (+ 381us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.932738 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:51:36.933043 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.922986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:41649)
0925 14:51:36.923224 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.923277 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.923817 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.924105 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.924110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.924727 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:36.924949 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.924954 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.925748 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.925749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.926685 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.926687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.927136 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.927137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.927272 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.927853 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:36.927864 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:36.929606 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:36.931341 (+ 1735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.931346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.931348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.931600 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.931846 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:36.931847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:36.931847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:36.931980 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:36.932260 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:36.932266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:36.932463 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.932599 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.932767 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":115,"threads_started":1}
I20260925 14:51:36.933589 15128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44123
I20260925 14:51:36.933693 15128 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:36.933863 15128 heartbeater.cc:511] Master 127.0.98.62:44123 requested a full tablet report, sending...
I20260925 14:51:36.934269 14669 ts_manager.cc:194] Registered new tserver with Master: 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
I20260925 14:51:36.934808 14669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41649
I20260925 14:51:36.935330 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.928401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.0.98.62:44123)
0925 14:51:36.928522 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.928526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.928544 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:51:36.928651 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.928652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.928666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.928743 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:36.929282 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.929816 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.930970 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.931172 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.931229 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.932395 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:36.932396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:36.932397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:36.932403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:36.934192 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.934564 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.934565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.934565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.934618 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:36.934796 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:36.934796 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:36.934796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:36.934935 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:36.935116 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.935159 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.935242 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":51,"threads_started":1}
I20260925 14:51:36.936180 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:51:36.938807 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.936513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37904 (local address 127.0.98.62:44123)
0925 14:51:36.936569 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.936571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.936660 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:51:36.936713 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.936713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.936740 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.936800 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:36.936884 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.937350 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.938029 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.938223 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.938280 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.938484 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:36.938582 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:36.938648 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.938742 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.938762 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:51:36.939157 14669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:36.940785 14669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:36.943199 14792 tablet_service.cc:1467] Processing CreateTablet for tablet 0b10958b78164ea2964a8a88627f98b3 (DEFAULT_TABLE table=test-table [id=6eedf6e593074044b7a9336f4cce88dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:36.943408 14792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b10958b78164ea2964a8a88627f98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:36.944942 14750 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Bootstrap starting.
I20260925 14:51:36.945302 14750 tablet_bootstrap.cc:654] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:36.945896 14750 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: No bootstrap required, opened a new log
I20260925 14:51:36.945943 14750 ts_tablet_manager.cc:1397] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:36.946285 14750 raft_consensus.cc:348] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:36.946350 14750 raft_consensus.cc:374] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:36.946379 14750 raft_consensus.cc:729] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4110f98847ef4f73b05035c45d306d29, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.946496 14750 consensus_queue.cc:260] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:36.946645 14750 ts_tablet_manager.cc:1428] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:36.947089 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.942769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42211 (local address 127.0.0.1:60784)
0925 14:51:36.942834 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.942846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.942889 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.943519 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.943520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.943522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.943570 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.943571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.944864 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.944865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.945903 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.945905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.946794 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.946795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.946878 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.946903 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.946975 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.947021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:51:36.947335 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.942882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.0.98.2:42211)
0925 14:51:36.943344 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.943348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.943358 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:51:36.943395 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.943397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.943411 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.943671 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.943741 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.944740 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.946027 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.946691 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.946804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.946869 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.947200 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.947249 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":77,"threads_started":1}
I20260925 14:51:36.948071 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.942769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46231 (local address 127.0.0.1:60104)
0925 14:51:36.942978 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:36.942988 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:36.943028 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:36.943237 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:36.943238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:36.943239 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:36.943273 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.943274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.944338 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.944339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.945661 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:36.945664 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.947807 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.947807 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:36.947916 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.947922 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:51:36.947966 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:51:36.947994 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":23,"thread_start_us":87,"threads_started":1}
I20260925 14:51:36.948158 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:36.942865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60104 (local address 127.0.98.3:46231)
0925 14:51:36.943071 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:36.943075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:36.943086 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:36.943122 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:36.943124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:36.943138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:36.943220 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:36.943377 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.944223 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.946857 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:36.947716 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:36.947800 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:36.947994 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:36.948025 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:51:36.948066 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":91,"threads_started":1}
I20260925 14:51:36.948109 14930 tablet_service.cc:1467] Processing CreateTablet for tablet 0b10958b78164ea2964a8a88627f98b3 (DEFAULT_TABLE table=test-table [id=6eedf6e593074044b7a9336f4cce88dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:36.948487 14930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b10958b78164ea2964a8a88627f98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:36.948724 15062 tablet_service.cc:1467] Processing CreateTablet for tablet 0b10958b78164ea2964a8a88627f98b3 (DEFAULT_TABLE table=test-table [id=6eedf6e593074044b7a9336f4cce88dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:36.948994 15062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b10958b78164ea2964a8a88627f98b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:36.951891 15138 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Bootstrap starting.
I20260925 14:51:36.951982 15139 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Bootstrap starting.
I20260925 14:51:36.952663 15139 tablet_bootstrap.cc:654] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:36.952775 15138 tablet_bootstrap.cc:654] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:36.953186 15139 log.cc:824] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.953219 15138 log.cc:824] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:36.953933 15138 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: No bootstrap required, opened a new log
I20260925 14:51:36.953933 15139 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: No bootstrap required, opened a new log
I20260925 14:51:36.954011 15139 ts_tablet_manager.cc:1397] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:36.954020 15138 ts_tablet_manager.cc:1397] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:36.955389 15138 raft_consensus.cc:348] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:36.955555 15138 raft_consensus.cc:374] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:36.955603 15138 raft_consensus.cc:729] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c57b36563604791bd6215a77faa3ddf, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.955550 15139 raft_consensus.cc:348] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:36.955755 15139 raft_consensus.cc:374] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:36.955726 15138 consensus_queue.cc:260] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:36.955802 15139 raft_consensus.cc:729] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669d40be33aa4a11a4188d2c95f39de2, State: Initialized, Role: FOLLOWER
I20260925 14:51:36.956007 15138 ts_tablet_manager.cc:1428] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:36.955992 15139 consensus_queue.cc:260] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:36.956091 15128 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
I20260925 14:51:36.956341 15139 ts_tablet_manager.cc:1428] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:51:36.956341 14996 heartbeater.cc:503] Master 127.0.98.62:44123 was elected leader, sending a full tablet report...
W20260925 14:51:37.019235 14997 tablet.cc:2369] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:37.135502 14863 raft_consensus.cc:482] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:37.135711 14863 raft_consensus.cc:504] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:37.136140 14863 leader_election.cc:288] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
I20260925 14:51:37.139226 15135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.136407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37109 (local address 127.0.98.2:42211)
0925 14:51:37.136451 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:37.136453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:37.136464 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:37.136556 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:37.136558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:37.136573 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:37.136640 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:37.136849 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.137386 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.138374 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.138805 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.138860 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:37.138868 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:37.139103 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 14:51:37.139147 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:51:37.139238 15136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.136432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45755 (local address 127.0.98.3:46231)
0925 14:51:37.136495 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:37.136498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:37.136630 (+ 132us) server_negotiation.cc:407] Connection header received
0925 14:51:37.136664 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:37.136665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:37.136678 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:37.136742 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:37.136996 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.137470 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.138461 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.138858 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.138932 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:37.138941 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:37.139130 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:51:37.139170 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:51:37.139245 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.136331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42211 (local address 127.0.98.1:37109)
0925 14:51:37.136388 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.136400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:37.136458 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:37.136661 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:37.136662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:37.136664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:37.136749 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:37.136751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.137470 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.137472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:37.138279 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:37.138281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.138889 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.138890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:37.138979 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:37.139009 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 14:51:37.139078 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:51:37.139147 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:51:37.139344 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.136337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46231 (local address 127.0.98.1:45755)
0925 14:51:37.136492 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.136502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:37.136589 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:37.136817 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:37.136818 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:37.136821 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:37.136906 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:37.136908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.137574 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.137574 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:37.138353 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:37.138356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.138953 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.138954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:37.139013 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:37.139017 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:37.139114 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:51:37.139150 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":18,"thread_start_us":61,"threads_started":1}
W20260925 14:51:37.139642 14859 tablet.cc:2369] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:37.139705 15082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10958b78164ea2964a8a88627f98b3" candidate_uuid: "4110f98847ef4f73b05035c45d306d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57b36563604791bd6215a77faa3ddf" is_pre_election: true
I20260925 14:51:37.139752 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10958b78164ea2964a8a88627f98b3" candidate_uuid: "4110f98847ef4f73b05035c45d306d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669d40be33aa4a11a4188d2c95f39de2" is_pre_election: true
I20260925 14:51:37.139849 15082 raft_consensus.cc:2454] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4110f98847ef4f73b05035c45d306d29 in term 0.
I20260925 14:51:37.139885 14950 raft_consensus.cc:2454] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4110f98847ef4f73b05035c45d306d29 in term 0.
I20260925 14:51:37.140111 14747 leader_election.cc:302] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29, 6c57b36563604791bd6215a77faa3ddf; no voters:
I20260925 14:51:37.140246 14863 raft_consensus.cc:2781] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:37.140306 14863 raft_consensus.cc:482] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:37.140322 14863 raft_consensus.cc:3037] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:37.141057 14863 raft_consensus.cc:504] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:37.141211 14863 leader_election.cc:288] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 1 election: Requested vote from peers 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
I20260925 14:51:37.141409 15082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10958b78164ea2964a8a88627f98b3" candidate_uuid: "4110f98847ef4f73b05035c45d306d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c57b36563604791bd6215a77faa3ddf"
I20260925 14:51:37.141443 14950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b10958b78164ea2964a8a88627f98b3" candidate_uuid: "4110f98847ef4f73b05035c45d306d29" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "669d40be33aa4a11a4188d2c95f39de2"
I20260925 14:51:37.141492 15082 raft_consensus.cc:3037] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:37.141527 14950 raft_consensus.cc:3037] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:37.142230 14950 raft_consensus.cc:2454] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4110f98847ef4f73b05035c45d306d29 in term 1.
I20260925 14:51:37.142234 15082 raft_consensus.cc:2454] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4110f98847ef4f73b05035c45d306d29 in term 1.
I20260925 14:51:37.142433 14747 leader_election.cc:302] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29, 6c57b36563604791bd6215a77faa3ddf; no voters:
I20260925 14:51:37.142609 14863 raft_consensus.cc:2781] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:37.142784 14863 raft_consensus.cc:686] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 LEADER]: Becoming Leader. State: Replica: 4110f98847ef4f73b05035c45d306d29, State: Running, Role: LEADER
I20260925 14:51:37.142861 14863 consensus_queue.cc:237] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:37.143692 14669 catalog_manager.cc:5495] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4110f98847ef4f73b05035c45d306d29 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4110f98847ef4f73b05035c45d306d29" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } health_report { overall_health: HEALTHY } } }
W20260925 14:51:37.172932 15129 tablet.cc:2369] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:37.175432 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.173143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.0.98.1:41737)
0925 14:51:37.173274 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:37.173275 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:37.173283 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:37.173317 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:37.173318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:37.173338 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:37.173409 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:37.173579 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.174043 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.174753 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:37.174935 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:37.174985 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:37.175116 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:37.175214 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:37.175268 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:37.175303 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:51:37.175319 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260925 14:51:37.176863 15082 raft_consensus.cc:1261] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Refusing update from remote peer 4110f98847ef4f73b05035c45d306d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:37.176882 14950 raft_consensus.cc:1261] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Refusing update from remote peer 4110f98847ef4f73b05035c45d306d29: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:37.177254 14863 consensus_queue.cc:1035] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:37.177397 15145 consensus_queue.cc:1035] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:37.179884 15151 mvcc.cc:205] Tried to move back new op lower bound from 7333264986833817600 to 7333264986697834496. Current Snapshot: MvccSnapshot[applied={T|T < 7333264986833817600}]
I20260925 14:51:37.180296 15150 mvcc.cc:205] Tried to move back new op lower bound from 7333264986833817600 to 7333264986697834496. Current Snapshot: MvccSnapshot[applied={T|T < 7333264986833817600}]
I20260925 14:51:37.182504 14668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:37.182621 14668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:37.183352 14668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Sending DeleteTablet for 3 replicas of tablet 0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:37.183677 14792 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b10958b78164ea2964a8a88627f98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45514
I20260925 14:51:37.183706 15062 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b10958b78164ea2964a8a88627f98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60104
I20260925 14:51:37.183708 14930 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b10958b78164ea2964a8a88627f98b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60784
I20260925 14:51:37.183928 15159 tablet_replica.cc:330] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: stopping tablet replica
I20260925 14:51:37.184033 15159 raft_consensus.cc:2229] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:37.184108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14708
I20260925 14:51:37.184149 15159 raft_consensus.cc:2258] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:37.184700 15159 ts_tablet_manager.cc:1905] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:37.185941 15160 tablet_replica.cc:330] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: stopping tablet replica
I20260925 14:51:37.186058 15160 raft_consensus.cc:2229] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:37.186164 15160 raft_consensus.cc:2258] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:37.186344 15159 ts_tablet_manager.cc:1918] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:37.186496 15159 log.cc:1196] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:37.186582 15160 ts_tablet_manager.cc:1905] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:37.186944 15159 ts_tablet_manager.cc:1939] T 0b10958b78164ea2964a8a88627f98b3 P 669d40be33aa4a11a4188d2c95f39de2: Deleting consensus metadata
I20260925 14:51:37.187470 14655 catalog_manager.cc:4841] TS 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211): tablet 0b10958b78164ea2964a8a88627f98b3 (table test-table [id=6eedf6e593074044b7a9336f4cce88dc]) successfully deleted
I20260925 14:51:37.188093 15160 ts_tablet_manager.cc:1918] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:37.188171 15160 log.cc:1196] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:37.188441 15160 ts_tablet_manager.cc:1939] T 0b10958b78164ea2964a8a88627f98b3 P 6c57b36563604791bd6215a77faa3ddf: Deleting consensus metadata
I20260925 14:51:37.188809 14656 catalog_manager.cc:4841] TS 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231): tablet 0b10958b78164ea2964a8a88627f98b3 (table test-table [id=6eedf6e593074044b7a9336f4cce88dc]) successfully deleted
W20260925 14:51:37.190925 14656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41737 (error 108)
I20260925 14:51:37.191149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14864
I20260925 14:51:37.191747 15134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.191306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41737 (local address 127.0.0.1:45520)
0925 14:51:37.191664 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.191704 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41737: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335}
W20260925 14:51:37.192057 14656 catalog_manager.cc:4584] TS 4110f98847ef4f73b05035c45d306d29 (127.0.98.1:41737): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b10958b78164ea2964a8a88627f98b3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41737: connect: Connection refused (error 111)
I20260925 14:51:37.196099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14999
I20260925 14:51:37.200508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 14636
I20260925 14:51:37.205690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--webserver_interface=127.0.98.62
--webserver_port=42977
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:37.287654 15162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:37.287904 15162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:37.287942 15162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:37.289232 15162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:37.289304 15162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:37.289328 15162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:37.289366 15162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:37.289391 15162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:37.290968 15162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42977
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:37.291251 15162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:37.291476 15162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:37.296344 15170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:37.296341 15168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:37.296344 15167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:37.296494 15162 server_base.cc:1034] running on GCE node
I20260925 14:51:37.296814 15162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:37.297303 15162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:37.298524 15162 hybrid_clock.cc:648] HybridClock initialized: now 1790347897298498 us; error 37 us; skew 500 ppm
Sep 25 14:51:37 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:37.301184 15162 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:37.301628 15162 webserver.cc:460] Webserver started at http://127.0.98.62:42977/ using document root <none> and password file <none>
I20260925 14:51:37.301865 15162 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:37.301913 15162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:37.303335 15162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:37.303985 15177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.304174 15162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:51:37.304283 15162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "662f737518dd4091a2a9ec59836b661c"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "24f911d1fc753fd6a2b90d489c3924b1"
server_key_iv: "0e252aef72fec27f84b9699d0531db5a"
server_key_version: "encryptionkey@0"
I20260925 14:51:37.304641 15162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:37.321636 15162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:37.324082 15162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:37.324249 15162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:37.328238 15162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44123
I20260925 14:51:37.328274 15229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44123 every 8 connection(s)
I20260925 14:51:37.328652 15162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:37.330137 15230 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:51:37.330292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15162
I20260925 14:51:37.330624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41737
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44123
--webserver_port=46323
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:37.331001 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap starting.
I20260925 14:51:37.339598 15230 log.cc:824] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:37.341853 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:51:37.342171 15230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c: Bootstrap complete.
I20260925 14:51:37.344425 15230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:37.344686 15230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 662f737518dd4091a2a9ec59836b661c, State: Initialized, Role: FOLLOWER
I20260925 14:51:37.344893 15230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:37.344997 15230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:37.345050 15230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:37.345098 15230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:51:37.346055 15230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:37.346174 15230 leader_election.cc:302] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 662f737518dd4091a2a9ec59836b661c; no voters:
I20260925 14:51:37.346374 15230 leader_election.cc:288] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:51:37.346427 15234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:51:37.346587 15234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [term 3 LEADER]: Becoming Leader. State: Replica: 662f737518dd4091a2a9ec59836b661c, State: Running, Role: LEADER
I20260925 14:51:37.346699 15230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:37.346870 15234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } }
I20260925 14:51:37.347473 15235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "662f737518dd4091a2a9ec59836b661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260925 14:51:37.347555 15235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:37.347530 15236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 662f737518dd4091a2a9ec59836b661c. Latest consensus state: current_term: 3 leader_uuid: "662f737518dd4091a2a9ec59836b661c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "662f737518dd4091a2a9ec59836b661c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44123 } } }
I20260925 14:51:37.347626 15236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 662f737518dd4091a2a9ec59836b661c [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:37.348026 15243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:37.348773 15243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:37.348851 15243 catalog_manager.cc:1259] Loaded cluster ID: 8a5256125f494d1aa3a70710d0d44155
I20260925 14:51:37.348910 15243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 14:51:37.349104 15243 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa44011d9ad google::LogMessage::Fail() at ??:0
@ 0x7fa44012088a google::LogMessage::SendToLog() at ??:0
@ 0x7fa44011d4de google::LogMessage::Flush() at ??:0
@ 0x7fa44011ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa44803e0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fa4480478b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa441feb0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa441fe0954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa442e446db start_thread at ??:0
@ 0x7fa43ec1b61f clone at ??:0
W20260925 14:51:37.427574 15232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:37.427776 15232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:37.427800 15232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:37.428970 15232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:37.429013 15232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:37.429041 15232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:37.430570 15232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41737
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46323
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:37.430820 15232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:37.431064 15232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:37.436293 15266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:37.436277 15267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:37.436452 15232 server_base.cc:1034] running on GCE node
W20260925 14:51:37.436271 15269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:37.436801 15232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:37.437081 15232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:37.438271 15232 hybrid_clock.cc:648] HybridClock initialized: now 1790347897438258 us; error 28 us; skew 500 ppm
Sep 25 14:51:37 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:37.441254 15232 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:37.441774 15232 webserver.cc:460] Webserver started at http://127.0.98.1:46323/ using document root <none> and password file <none>
I20260925 14:51:37.442039 15232 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:37.442094 15232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:37.443845 15232 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:37.444639 15276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.444830 15232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:37.444936 15232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "4110f98847ef4f73b05035c45d306d29"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "602ac2567d0573c43404b7e76b9c1596"
server_key_iv: "d30a9956f9e733d82d9e95d753e4eede"
server_key_version: "encryptionkey@0"
I20260925 14:51:37.445314 15232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:37.468750 15232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:37.471221 15232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:37.471371 15232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:37.471637 15232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:37.472110 15283 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:51:37.474020 15232 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:51:37.474089 15232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:51:37.474136 15232 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:51:37.474798 15232 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:51:37.474855 15232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:51:37.474920 15283 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Bootstrap starting.
I20260925 14:51:37.481451 15232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41737
I20260925 14:51:37.481819 15232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:37.485634 15283 log.cc:824] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:37.485908 15390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41737 every 8 connection(s)
I20260925 14:51:37.487794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15232
I20260925 14:51:37.487848 15283 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:51:37.488066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42211
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44123
--webserver_port=44687
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:51:37.488134 15283 tablet_bootstrap.cc:492] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Bootstrap complete.
I20260925 14:51:37.489209 15283 ts_tablet_manager.cc:1397] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20260925 14:51:37.490775 15283 raft_consensus.cc:348] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:37.491036 15283 raft_consensus.cc:729] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4110f98847ef4f73b05035c45d306d29, State: Initialized, Role: FOLLOWER
I20260925 14:51:37.491216 15283 consensus_queue.cc:260] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:37.491503 15283 ts_tablet_manager.cc:1428] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260925 14:51:37.565701 15393 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:51:37.482585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:45811)
0925 14:51:37.482790 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.482827 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:37.483260 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:37.565503 (+ 82243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":122,"threads_started":1}
W20260925 14:51:37.565994 15391 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44123 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44123: Client connection negotiation failed: client connection to 127.0.98.62:44123: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:51:37.566375 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.566259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:47341)
0925 14:51:37.566313 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.566356 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:51:37.566762 15393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.566675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:58959)
0925 14:51:37.566723 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.566747 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:51:37.566895 15391 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:51:37.588223 15395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:37.588503 15395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:37.588543 15395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:37.589833 15395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:37.589906 15395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:37.589953 15395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:37.591547 15395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42211
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44687
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:37.591828 15395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:37.592062 15395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:37.597111 15402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:37.597158 15401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:37.597319 15395 server_base.cc:1034] running on GCE node
W20260925 14:51:37.597332 15404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:37.597661 15395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:37.597905 15395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:37.599076 15395 hybrid_clock.cc:648] HybridClock initialized: now 1790347897599078 us; error 39 us; skew 500 ppm
Sep 25 14:51:37 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:37.601665 15395 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:37.602118 15395 webserver.cc:460] Webserver started at http://127.0.98.2:44687/ using document root <none> and password file <none>
I20260925 14:51:37.602357 15395 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:37.602461 15395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:37.603855 15395 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:37.604535 15411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.604749 15395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:37.604864 15395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "669d40be33aa4a11a4188d2c95f39de2"
format_stamp: "Formatted at 2026-09-25 14:51:35 on dist-test-slave-z2s5"
server_key: "f6449ced5c79ee9ff60dd995ddbfe3f2"
server_key_iv: "7e06bf4879ceb6d2e3f25ce5b183b154"
server_key_version: "encryptionkey@0"
I20260925 14:51:37.605239 15395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:37.631111 15395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:37.633220 15395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:37.633383 15395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:37.633636 15395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:37.633939 15395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:37.633992 15395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.634034 15395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:37.634059 15395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.640084 15395 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42211
I20260925 14:51:37.640125 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42211 every 8 connection(s)
I20260925 14:51:37.640509 15395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:51:37.641566 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.641199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:45535)
0925 14:51:37.641353 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.641425 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
W20260925 14:51:37.641974 15525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44123 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44123: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
I20260925 14:51:37.642236 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.642151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:51629)
0925 14:51:37.642194 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.642220 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:51:37.642551 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.642481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:42295)
0925 14:51:37.642518 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.642539 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:51:37.642661 15525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:51:37.644858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15395
I20260925 14:51:37.645099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46231
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44123
--webserver_port=34981
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:37.724895 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:37.725136 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:37.725162 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:37.726245 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:37.726279 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:37.726305 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:37.727792 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46231
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34981
--tserver_master_addrs=127.0.98.62:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:37.727972 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:37.728150 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:37.733237 15533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:37.733242 15536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:37.733273 15534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:37.733670 15528 server_base.cc:1034] running on GCE node
I20260925 14:51:37.733928 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:37.734174 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:37.735361 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790347897735328 us; error 64 us; skew 500 ppm
Sep 25 14:51:37 dist-test-slave-z2s5 krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:37.738109 15528 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:37.738691 15528 webserver.cc:460] Webserver started at http://127.0.98.3:34981/ using document root <none> and password file <none>
I20260925 14:51:37.738955 15528 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:37.739037 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:37.740504 15528 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:51:37.741236 15543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.741429 15528 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:51:37.741534 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "6c57b36563604791bd6215a77faa3ddf"
format_stamp: "Formatted at 2026-09-25 14:51:36 on dist-test-slave-z2s5"
server_key: "69b58e948a554c7813fa05ca9a970dc4"
server_key_iv: "2a9105c3bf03d1d8f6deff26248952b3"
server_key_version: "encryptionkey@0"
I20260925 14:51:37.741911 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:37.755024 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:37.757241 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:37.757416 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:37.757673 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:37.757989 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:37.758050 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.758087 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:37.758136 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:37.764253 15528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46231
I20260925 14:51:37.764302 15656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46231 every 8 connection(s)
I20260925 14:51:37.764645 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:37.765733 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.765311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:47201)
0925 14:51:37.765492 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.765593 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
W20260925 14:51:37.766201 15657 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44123 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44123: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
I20260925 14:51:37.766562 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.766384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:57467)
0925 14:51:37.766496 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.766531 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 14:51:37.766891 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:37.766807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:59225)
0925 14:51:37.766851 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:37.766875 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:51:37.767048 15657 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:51:37.770123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15528
I20260925 14:51:38.567539 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.567295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:58077)
0925 14:51:38.567424 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:38.567474 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260925 14:51:38.617116 15662 raft_consensus.cc:482] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:38.617239 15662 raft_consensus.cc:504] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:38.617534 15662 leader_election.cc:288] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
Sep 25 14:51:38 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 14:51:38 dist-test-slave-z2s5 krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 14:51:38.629447 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.617850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45713 (local address 127.0.98.2:42211)
0925 14:51:38.618022 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:38.618033 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:38.618057 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:51:38.618626 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:38.618628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:38.618702 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:38.618848 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:38.619708 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.620655 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.621688 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.622091 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.622226 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:38.624815 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:38.624819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:38.624819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:38.624830 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:38.627029 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:38.627727 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:38.627730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:38.627730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:38.627857 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:38.628291 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:38.628292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:38.628292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:38.628636 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:38.628724 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:38.629180 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 14:51:38.629321 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":56,"threads_started":1}
I20260925 14:51:38.629436 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.617870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34627 (local address 127.0.98.3:46231)
0925 14:51:38.618009 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:38.618023 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:38.618049 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:51:38.618572 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:38.618576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:38.618650 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:38.618821 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:38.619713 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.620621 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.621672 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.622068 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.622185 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:38.625246 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:38.625249 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:38.625249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:38.625260 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:38.627234 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:38.627727 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:38.627730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:38.627730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:38.627857 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:38.628291 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:38.628291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:38.628292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:38.628636 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:38.628724 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:38.629180 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 14:51:38.629321 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260925 14:51:38.629601 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.617745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46231 (local address 127.0.98.1:34627)
0925 14:51:38.617795 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:38.617806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:38.617861 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:38.618851 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:38.618854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:38.619286 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:38.619554 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:38.619562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.620736 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.620737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:38.621566 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:38.621568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.622170 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.622171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:38.622296 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:38.623082 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:38.623095 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:38.625146 (+ 2051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:38.627322 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:38.627322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:38.627323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:38.627630 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:38.627996 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:38.627997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:38.627998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:38.628207 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:38.628744 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:38.628753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:38.628936 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 14:51:38.629106 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 14:51:38.629334 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":11}
I20260925 14:51:38.629784 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.617752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42211 (local address 127.0.98.1:45713)
0925 14:51:38.617938 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:38.617947 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:38.617988 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:38.618929 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:38.618929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:38.619295 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:38.619554 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:38.619562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.620784 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.620785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:38.621565 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:38.621568 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:38.622195 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:38.622196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:38.622291 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:38.623071 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:38.623080 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:38.624703 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:38.627137 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:38.627143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:38.627145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:38.627630 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:38.627996 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:38.627997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:38.627997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:38.628207 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:38.628744 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:38.628753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:38.628936 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 14:51:38.629106 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 14:51:38.629677 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":434,"thread_start_us":82,"threads_started":1}
W20260925 14:51:38.630664 15280 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
W20260925 14:51:38.630945 15279 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:38.631006 15279 leader_election.cc:302] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29; no voters: 669d40be33aa4a11a4188d2c95f39de2, 6c57b36563604791bd6215a77faa3ddf
I20260925 14:51:38.631130 15662 raft_consensus.cc:2726] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:51:38.643225 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.643015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:46307)
0925 14:51:38.643145 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:38.643186 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260925 14:51:38.767865 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:38.767504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:46601)
0925 14:51:38.767694 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:38.767769 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":66,"threads_started":1}
I20260925 14:51:39.568435 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:39.568149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:44085)
0925 14:51:39.568310 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:39.568365 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260925 14:51:39.643954 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:39.643735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:43127)
0925 14:51:39.643858 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:39.643894 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260925 14:51:39.768909 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:39.768625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:55329)
0925 14:51:39.768809 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:39.768857 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
I20260925 14:51:40.550539 15671 raft_consensus.cc:482] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:40.550631 15671 raft_consensus.cc:504] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:40.550812 15671 leader_election.cc:288] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
W20260925 14:51:40.551206 15279 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
W20260925 14:51:40.551285 15280 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:40.551378 15280 leader_election.cc:302] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29; no voters: 669d40be33aa4a11a4188d2c95f39de2, 6c57b36563604791bd6215a77faa3ddf
I20260925 14:51:40.551466 15671 raft_consensus.cc:2726] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:51:40.569139 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:40.568885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:55971)
0925 14:51:40.569045 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:40.569090 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
I20260925 14:51:40.644688 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:40.644458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:50527)
0925 14:51:40.644594 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:40.644641 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":72,"threads_started":1}
I20260925 14:51:40.769775 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:40.769533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:39803)
0925 14:51:40.769696 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:40.769733 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
I20260925 14:51:41.570043 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:41.569766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:48383)
0925 14:51:41.569943 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:41.569986 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":83,"threads_started":1}
I20260925 14:51:41.645354 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:41.645150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:35849)
0925 14:51:41.645270 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:41.645310 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260925 14:51:41.770539 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:41.770270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:47507)
0925 14:51:41.770461 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:41.770496 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":63,"threads_started":1}
I20260925 14:51:42.570897 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:42.570651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:33769)
0925 14:51:42.570819 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:42.570851 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260925 14:51:42.646067 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:42.645808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:36927)
0925 14:51:42.645980 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:42.646019 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
I20260925 14:51:42.771181 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:42.770976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:52233)
0925 14:51:42.771107 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:42.771139 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260925 14:51:43.571733 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:43.571483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:33403)
0925 14:51:43.571651 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:43.571688 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260925 14:51:43.646871 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:43.646612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:38943)
0925 14:51:43.646785 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:43.646824 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260925 14:51:43.674589 15683 raft_consensus.cc:482] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:43.674707 15683 raft_consensus.cc:504] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:43.674911 15683 leader_election.cc:288] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
W20260925 14:51:43.675369 15280 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
W20260925 14:51:43.675449 15279 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:43.675495 15279 leader_election.cc:302] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29; no voters: 669d40be33aa4a11a4188d2c95f39de2, 6c57b36563604791bd6215a77faa3ddf
I20260925 14:51:43.675583 15683 raft_consensus.cc:2726] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:51:43.771906 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:43.771668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:59665)
0925 14:51:43.771818 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:43.771861 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
I20260925 14:51:44.572628 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:44.572349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:36215)
0925 14:51:44.572518 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:44.572568 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":66,"threads_started":1}
I20260925 14:51:44.647768 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:44.647508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:33201)
0925 14:51:44.647659 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:44.647701 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":80,"threads_started":1}
I20260925 14:51:44.772665 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:44.772448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:57147)
0925 14:51:44.772582 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:44.772621 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260925 14:51:45.573474 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:45.573251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:40301)
0925 14:51:45.573384 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:45.573428 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260925 14:51:45.648512 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:45.648223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:33177)
0925 14:51:45.648375 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:45.648452 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260925 14:51:45.773453 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:45.773237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:57077)
0925 14:51:45.773371 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:45.773412 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260925 14:51:46.574144 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:46.573946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:43523)
0925 14:51:46.574066 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:46.574099 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260925 14:51:46.649118 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:46.648910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:55055)
0925 14:51:46.649037 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:46.649073 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260925 14:51:46.774091 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:46.773900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:47531)
0925 14:51:46.774014 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:46.774049 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260925 14:51:47.574966 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:47.574767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:54083)
0925 14:51:47.574885 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:47.574920 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260925 14:51:47.649852 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:47.649636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:57501)
0925 14:51:47.649754 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:47.649806 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260925 14:51:47.774773 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:47.774562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:34935)
0925 14:51:47.774693 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:47.774728 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260925 14:51:47.807073 15697 raft_consensus.cc:482] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:47.807154 15697 raft_consensus.cc:504] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "669d40be33aa4a11a4188d2c95f39de2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42211 } } peers { permanent_uuid: "6c57b36563604791bd6215a77faa3ddf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46231 } } peers { permanent_uuid: "4110f98847ef4f73b05035c45d306d29" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41737 } }
I20260925 14:51:47.807423 15697 leader_election.cc:288] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211), 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231)
W20260925 14:51:47.807698 15279 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 669d40be33aa4a11a4188d2c95f39de2 (127.0.98.2:42211): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
W20260925 14:51:47.807861 15280 leader_election.cc:341] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c57b36563604791bd6215a77faa3ddf (127.0.98.3:46231): Not found: Tablet not found: 0b10958b78164ea2964a8a88627f98b3
I20260925 14:51:47.807928 15280 leader_election.cc:302] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4110f98847ef4f73b05035c45d306d29; no voters: 669d40be33aa4a11a4188d2c95f39de2, 6c57b36563604791bd6215a77faa3ddf
I20260925 14:51:47.808032 15697 raft_consensus.cc:2726] T 0b10958b78164ea2964a8a88627f98b3 P 4110f98847ef4f73b05035c45d306d29 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:51:48.575637 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:48.575432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:39707)
0925 14:51:48.575553 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:48.575591 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260925 14:51:48.650557 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:48.650320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:58057)
0925 14:51:48.650465 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:48.650513 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":121,"threads_started":1}
I20260925 14:51:48.775446 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:48.775208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:51745)
0925 14:51:48.775356 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:48.775399 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260925 14:51:49.576366 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:49.576123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:55397)
0925 14:51:49.576279 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:49.576321 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":66,"threads_started":1}
I20260925 14:51:49.651211 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:49.650976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:35077)
0925 14:51:49.651120 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:49.651166 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":57,"threads_started":1}
I20260925 14:51:49.776196 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:49.775958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:57885)
0925 14:51:49.776109 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:49.776148 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":86,"threads_started":1}
I20260925 14:51:50.577112 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:50.576879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:47005)
0925 14:51:50.577032 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:50.577066 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260925 14:51:50.651870 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:50.651640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:41411)
0925 14:51:50.651778 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:50.651822 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260925 14:51:50.777007 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:50.776728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:36907)
0925 14:51:50.776916 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:50.776962 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260925 14:51:51.577868 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:51.577626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:39373)
0925 14:51:51.577777 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:51.577820 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260925 14:51:51.652593 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:51.652388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:32841)
0925 14:51:51.652508 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:51.652549 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260925 14:51:51.777810 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:51.777574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:55047)
0925 14:51:51.777710 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:51.777766 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260925 14:51:52.578812 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:52.578595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.1:58919)
0925 14:51:52.578727 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:52.578768 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260925 14:51:52.653386 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:52.653167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.2:44689)
0925 14:51:52.653294 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:52.653336 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 14:51:52.778748 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:52.778522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44123 (local address 127.0.98.3:38651)
0925 14:51:52.778658 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:52.778703 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":65,"threads_started":1}
I20260925 14:51:52.781354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 15232
I20260925 14:51:52.786031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 15395
I20260925 14:51:52.790489 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 15528
2026-09-25T14:51:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18960 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:51:53 dist-test-slave-z2s5 krb5kdc[15715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:51:53 dist-test-slave-z2s5 krb5kdc[15715](info): set up 2 sockets
Sep 25 14:51:53 dist-test-slave-z2s5 krb5kdc[15715](info): commencing operation
krb5kdc: starting...
W20260925 14:51:54.366665 392 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:51:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:51:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:51:54.490656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:54.570698 15728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:54.570967 15728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:54.571005 15728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:54.572252 15728 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:51:54.572326 15728 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:51:54.572350 15728 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:51:54.572366 15728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:51:54.572387 15728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:54.572419 15728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:54.572444 15728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:51:54.572472 15728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:51:54.573978 15728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:54.574260 15728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:54.574523 15728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:51:54.579797 15728 server_base.cc:1034] running on GCE node
W20260925 14:51:54.579777 15733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:54.579772 15736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:54.579785 15734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:54.580289 15728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:54.580586 15728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:54.581756 15728 hybrid_clock.cc:648] HybridClock initialized: now 1790347914581731 us; error 46 us; skew 500 ppm
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:54.584386 15728 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:51:54.584851 15728 webserver.cc:460] Webserver started at http://127.0.98.62:44625/ using document root <none> and password file <none>
I20260925 14:51:54.585079 15728 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:54.585146 15728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:54.585299 15728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:54.586472 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "4c0467d46758467fb61de3ee87367a2e"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "cad28d15f3428b70a145576bb279ea18"
server_key_iv: "d2999a81721005667fbce5318604355c"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.586850 15728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "4c0467d46758467fb61de3ee87367a2e"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "cad28d15f3428b70a145576bb279ea18"
server_key_iv: "d2999a81721005667fbce5318604355c"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.588294 15728 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:51:54.589133 15743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.589397 15728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:54.589501 15728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "4c0467d46758467fb61de3ee87367a2e"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "cad28d15f3428b70a145576bb279ea18"
server_key_iv: "d2999a81721005667fbce5318604355c"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.589592 15728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:54.608907 15728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:54.611002 15728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:54.611171 15728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:54.615132 15728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35681
I20260925 14:51:54.615171 15795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35681 every 8 connection(s)
I20260925 14:51:54.615573 15728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:51:54.616545 15796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:54.618857 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap starting.
I20260925 14:51:54.619391 15796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:54.619642 15796 log.cc:824] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:54.620350 15796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: No bootstrap required, opened a new log
I20260925 14:51:54.621932 15796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:51:54.622071 15796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:54.622094 15796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0467d46758467fb61de3ee87367a2e, State: Initialized, Role: FOLLOWER
I20260925 14:51:54.622237 15796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:51:54.622331 15796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:51:54.622355 15796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:51:54.622383 15796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:54.622920 15796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:51:54.623010 15796 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0467d46758467fb61de3ee87367a2e; no voters:
I20260925 14:51:54.623135 15796 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:51:54.623214 15799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:54.623401 15799 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 LEADER]: Becoming Leader. State: Replica: 4c0467d46758467fb61de3ee87367a2e, State: Running, Role: LEADER
I20260925 14:51:54.623432 15796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:51:54.623505 15799 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:51:54.623888 15800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0467d46758467fb61de3ee87367a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260925 14:51:54.623988 15800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:54.623898 15801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0467d46758467fb61de3ee87367a2e. Latest consensus state: current_term: 1 leader_uuid: "4c0467d46758467fb61de3ee87367a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260925 14:51:54.624078 15801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:51:54.624507 15805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:51:54.625012 15805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:51:54.625694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15728
I20260925 14:51:54.625815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:51:54.626024 392 external_mini_cluster.cc:1428] Setting key e0f8a73fd968a15a8b6f7d419853c032
I20260925 14:51:54.627036 15805 catalog_manager.cc:1347] Generated new cluster ID: 9193d852cd57483a8eec21b5431a536a
I20260925 14:51:54.627107 15805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:54.638092 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:54.626599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.0.98.62:35681)
0925 14:51:54.627300 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:54.627310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:54.627354 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:51:54.627483 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:54.627490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:54.627544 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:54.627683 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:54.628386 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.629847 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.630896 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.631539 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.631652 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:54.634459 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:54.634464 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:54.634464 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:54.634476 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:54.636438 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:54.636870 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:54.636873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:54.636873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:54.636973 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:54.637180 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:54.637180 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:54.637181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:54.637472 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:54.637600 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:54.637730 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:51:54.637859 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":659,"thread_start_us":71,"threads_started":1}
I20260925 14:51:54.645949 15805 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:51:54.646589 15805 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:51:54.648618 15805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Generated new TSK 0
I20260925 14:51:54.648808 15805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:51:54.675297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:54.756350 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:54.756579 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:54.756603 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:54.757737 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:54.757781 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:54.757807 15822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:51:54.759310 15822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:54.759559 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:54.759771 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:54.764747 15827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:54.764747 15830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:54.764763 15828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:54.765086 15822 server_base.cc:1034] running on GCE node
I20260925 14:51:54.765300 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:54.765542 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:54.766697 15822 hybrid_clock.cc:648] HybridClock initialized: now 1790347914766687 us; error 32 us; skew 500 ppm
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:54.769166 15822 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:51:54.769614 15822 webserver.cc:460] Webserver started at http://127.0.98.1:46031/ using document root <none> and password file <none>
I20260925 14:51:54.769848 15822 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:54.769920 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:54.770052 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:54.771147 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "dba6367128f94bbbaebdc4795dbb59c2"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "7fde3e8e98bcf01812166df18875960b"
server_key_iv: "e1a3e4f94b18a008a714a4f742f20abe"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.771503 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dba6367128f94bbbaebdc4795dbb59c2"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "7fde3e8e98bcf01812166df18875960b"
server_key_iv: "e1a3e4f94b18a008a714a4f742f20abe"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.772857 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:51:54.773612 15837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.773793 15822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:54.773896 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "dba6367128f94bbbaebdc4795dbb59c2"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "7fde3e8e98bcf01812166df18875960b"
server_key_iv: "e1a3e4f94b18a008a714a4f742f20abe"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.773975 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:54.797575 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:54.799598 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:54.799769 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:54.800014 15822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:54.800346 15822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:54.800401 15822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.800442 15822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:54.800470 15822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.806411 15822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40659
I20260925 14:51:54.806483 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40659 every 8 connection(s)
I20260925 14:51:54.806814 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:51:54.810333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15822
I20260925 14:51:54.810467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:51:54.810655 392 external_mini_cluster.cc:1428] Setting key 55f414a4b296da32383c47dba25fbc21
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:54.816473 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:54.807554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52579 (local address 127.0.98.62:35681)
0925 14:51:54.807620 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:54.807622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:54.807812 (+ 190us) server_negotiation.cc:407] Connection header received
0925 14:51:54.808284 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:54.808285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:54.808295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:54.808352 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:54.809235 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.809762 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.810927 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.811149 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.811233 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:54.813327 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:54.813328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:54.813328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:54.813336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:54.814844 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:54.815333 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:54.815333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:54.815334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:54.815413 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:54.815709 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:54.815709 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:54.815710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:54.815809 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:54.815856 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:54.816306 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 14:51:54.816399 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:51:54.816684 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:54.807529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:52579)
0925 14:51:54.807690 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:54.807730 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:54.808204 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:54.808407 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:54.808411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:54.808966 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:54.809158 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:54.809162 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.809860 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.809861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:54.810841 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:54.810844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.811244 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.811245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:54.811378 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:54.811909 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:54.811917 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:54.813244 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:54.814922 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:54.814929 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:54.814931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:54.815255 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:54.815482 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:54.815482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:54.815483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:54.815635 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:54.815873 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:54.815879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:54.816059 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 14:51:54.816191 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 14:51:54.816377 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
I20260925 14:51:54.817265 15951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260925 14:51:54.817399 15951 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:54.817600 15951 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260925 14:51:54.818233 15760 ts_manager.cc:194] Registered new tserver with Master: dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659)
I20260925 14:51:54.818995 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52579
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:51:54.835311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:54.915192 15956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:54.915390 15956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:54.915412 15956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:54.916551 15956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:54.916594 15956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:54.916648 15956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:51:54.918048 15956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:54.918285 15956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:54.918547 15956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:54.923434 15962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:54.923470 15956 server_base.cc:1034] running on GCE node
W20260925 14:51:54.923439 15961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:54.923532 15964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:54.923851 15956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:54.924116 15956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:54.925249 15956 hybrid_clock.cc:648] HybridClock initialized: now 1790347914925240 us; error 27 us; skew 500 ppm
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:54.927803 15956 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:51:54.928265 15956 webserver.cc:460] Webserver started at http://127.0.98.2:42613/ using document root <none> and password file <none>
I20260925 14:51:54.928483 15956 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:54.928527 15956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:54.928674 15956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:54.929745 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "543a507efe568b01b86bd92e406f3d25"
server_key_iv: "2e6c0186d42d850657fe13f874d82b28"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.930097 15956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "543a507efe568b01b86bd92e406f3d25"
server_key_iv: "2e6c0186d42d850657fe13f874d82b28"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.931443 15956 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:51:54.932097 15971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.932263 15956 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:51:54.932344 15956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "543a507efe568b01b86bd92e406f3d25"
server_key_iv: "2e6c0186d42d850657fe13f874d82b28"
server_key_version: "encryptionkey@0"
I20260925 14:51:54.932399 15956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:54.944540 15956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:54.946754 15956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:54.946863 15956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:54.947036 15956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:54.947273 15956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:54.947302 15956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.947322 15956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:54.947335 15956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:54.952870 15956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45995
I20260925 14:51:54.952973 16084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45995 every 8 connection(s)
I20260925 14:51:54.953238 15956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:51:54 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:54.960120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 15956
I20260925 14:51:54.960238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:51:54.960475 392 external_mini_cluster.cc:1428] Setting key 7e107a54d47ca12b9241f3046a45170f
I20260925 14:51:54.962586 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:54.953960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45935 (local address 127.0.98.62:35681)
0925 14:51:54.954006 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:54.954008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:54.954196 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:51:54.954780 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:54.954781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:54.954791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:54.954850 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:54.955767 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.956263 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.957197 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.957379 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.957431 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:54.959579 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:54.959580 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:54.959580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:54.959587 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:54.961188 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:54.961609 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:54.961610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:54.961611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:54.961673 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:54.961925 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:54.961926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:54.961926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:54.962039 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:54.962109 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:54.962378 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:51:54.962488 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:51:54.962803 16087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:54.953907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:45935)
0925 14:51:54.954074 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:54.954115 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:54.954691 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:54.954935 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:54.954938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:54.955495 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:54.955683 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:54.955688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.956341 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.956341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:54.957125 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:54.957127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:54.957450 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:54.957450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:54.957543 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:54.958093 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:54.958110 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:54.959495 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:54.961280 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:54.961287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:54.961289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:54.961515 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:54.961742 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:54.961743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:54.961743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:54.961850 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:54.962112 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:54.962119 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:54.962252 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:51:54.962357 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:51:54.962578 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260925 14:51:54.963375 16085 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260925 14:51:54.963459 16085 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:54.963645 16085 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260925 14:51:54.964015 15760 ts_manager.cc:194] Registered new tserver with Master: b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
I20260925 14:51:54.964396 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45935
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:51:54.985731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:51:55.067268 16090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:51:55.067489 16090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:51:55.067514 16090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:51:55.068709 16090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:51:55.068758 16090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:51:55.068785 16090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:51:55.070370 16090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:51:55.070653 16090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:51:55.070904 16090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:51:55.075996 16095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:51:55.076036 16096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:55.076046 16090 server_base.cc:1034] running on GCE node
W20260925 14:51:55.076040 16098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:51:55.076458 16090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:51:55.076702 16090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:51:55.077873 16090 hybrid_clock.cc:648] HybridClock initialized: now 1790347915077872 us; error 41 us; skew 500 ppm
Sep 25 14:51:55 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:51:55.080623 16090 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:51:55.081053 16090 webserver.cc:460] Webserver started at http://127.0.98.3:40065/ using document root <none> and password file <none>
I20260925 14:51:55.081257 16090 fs_manager.cc:359] Metadata directory not provided
I20260925 14:51:55.081308 16090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:51:55.081444 16090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:51:55.082688 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "7a0fa856dc5347b9b63a2c3de72dfada"
format_stamp: "Formatted at 2026-09-25 14:51:55 on dist-test-slave-z2s5"
server_key: "a6855c98fa76585470b989555d6eec2a"
server_key_iv: "e13f1ff88a26f64af8dfe4848f79dc8e"
server_key_version: "encryptionkey@0"
I20260925 14:51:55.083017 16090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7a0fa856dc5347b9b63a2c3de72dfada"
format_stamp: "Formatted at 2026-09-25 14:51:55 on dist-test-slave-z2s5"
server_key: "a6855c98fa76585470b989555d6eec2a"
server_key_iv: "e13f1ff88a26f64af8dfe4848f79dc8e"
server_key_version: "encryptionkey@0"
I20260925 14:51:55.084465 16090 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:51:55.085566 16105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:55.085719 16090 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:51:55.085808 16090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "7a0fa856dc5347b9b63a2c3de72dfada"
format_stamp: "Formatted at 2026-09-25 14:51:55 on dist-test-slave-z2s5"
server_key: "a6855c98fa76585470b989555d6eec2a"
server_key_iv: "e13f1ff88a26f64af8dfe4848f79dc8e"
server_key_version: "encryptionkey@0"
I20260925 14:51:55.085877 16090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:51:55.095899 16090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:51:55.098157 16090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:51:55.098266 16090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:51:55.098511 16090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:51:55.098788 16090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:51:55.098824 16090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:55.098855 16090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:51:55.098867 16090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:51:55.104453 16090 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44421
I20260925 14:51:55.104530 16218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44421 every 8 connection(s)
I20260925 14:51:55.104854 16090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:51:55.110682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16090
I20260925 14:51:55.110780 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:51:55.110991 392 external_mini_cluster.cc:1428] Setting key 8caf76b2d05c727e5a93a37f7744c600
Sep 25 14:51:55 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:51:55.115195 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.105545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40475 (local address 127.0.98.62:35681)
0925 14:51:55.105590 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.105592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.105834 (+ 242us) server_negotiation.cc:407] Connection header received
0925 14:51:55.106602 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.106603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.106628 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.106715 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:55.107927 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.108412 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.109456 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.109633 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.109686 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.112370 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:55.112371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:55.112372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:55.112378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:55.113886 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:55.114330 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:55.114331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:55.114331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:55.114448 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:55.114675 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:55.114675 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:55.114676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:55.114791 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:55.114861 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.115066 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.115133 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:51:55.115439 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.105519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:40475)
0925 14:51:55.105683 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.105730 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:55.106327 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:55.107010 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:55.107013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:55.107613 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:51:55.107810 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.107814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.108539 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.108540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.109371 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.109373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.109710 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.109711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.109830 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.110369 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:51:55.110377 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:51:55.112273 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:51:55.113978 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:55.113983 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:55.113985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:55.114231 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:55.114515 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:51:55.114515 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:51:55.114516 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:51:55.114615 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:51:55.114851 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:51:55.114855 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:51:55.114973 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 14:51:55.115054 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:51:55.115174 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260925 14:51:55.115948 16219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260925 14:51:55.116029 16219 heartbeater.cc:464] Registering TS with master...
I20260925 14:51:55.116227 16219 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260925 14:51:55.116669 15760 ts_manager.cc:194] Registered new tserver with Master: 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421)
I20260925 14:51:55.117056 15760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40475
I20260925 14:51:55.123003 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:51:55.131446 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.124616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.98.62:35681)
0925 14:51:55.124665 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.124667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.124742 (+ 75us) server_negotiation.cc:407] Connection header received
0925 14:51:55.124857 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.124858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.124868 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.124919 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:51:55.125539 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.126040 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.126991 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.127223 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.127309 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.128565 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:51:55.128566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:51:55.128567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:51:55.128574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:51:55.129984 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:55.130490 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:55.130491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:55.130491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:55.130590 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:51:55.130830 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:51:55.130831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:51:55.130831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:51:55.131001 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:51:55.131120 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.131246 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.131331 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:51:55.133211 15760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:51:55.134999 15760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:51:55.141549 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.137352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45995 (local address 127.0.0.1:43408)
0925 14:51:55.137918 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.137926 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:55.137962 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:55.138385 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:55.138386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:55.138407 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:55.138439 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.138440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.139653 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.139654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.140540 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.140543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.141265 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.141266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.141341 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.141355 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:51:55.141427 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:51:55.141477 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":52,"threads_started":1}
I20260925 14:51:55.141564 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.137823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.0.98.2:45995)
0925 14:51:55.138196 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.138201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.138212 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:51:55.138266 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.138269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.138286 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.138380 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:55.138497 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.139531 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.140657 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.141168 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.141254 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.141327 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.141443 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.141483 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
I20260925 14:51:55.141798 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.137334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44421 (local address 127.0.0.1:40032)
0925 14:51:55.137776 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.137787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:55.137847 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:55.138300 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:55.138301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:55.138302 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:55.138330 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.138331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.139239 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.139240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.140518 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.140521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.141563 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.141564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.141632 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.141635 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:55.141687 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:51:55.141723 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":89,"threads_started":1}
I20260925 14:51:55.142019 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.137324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.0.1:52552)
0925 14:51:55.137476 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.137517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:55.137591 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:55.138065 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:55.138069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:55.138071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:55.138142 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.138145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.139490 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.139491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.140649 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.140651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.141494 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.141495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.141576 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.141580 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:55.141616 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 14:51:55.141957 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":43,"thread_start_us":76,"threads_started":1}
I20260925 14:51:55.142263 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.137588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40032 (local address 127.0.98.3:44421)
0925 14:51:55.137762 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.137768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.138089 (+ 321us) server_negotiation.cc:407] Connection header received
0925 14:51:55.138133 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.138135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.138151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.138224 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:55.138441 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.139142 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.140614 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.141376 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.141460 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.141530 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.141987 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.142150 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260925 14:51:55.142596 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.137603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.0.98.1:40659)
0925 14:51:55.137848 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.137853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.137867 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:51:55.137913 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.137915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.137931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.138056 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:55.138211 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.139382 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.140711 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.141386 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.142361 (+ 975us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.142456 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.142483 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.142527 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260925 14:51:55.142490 16019 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc82b5eff640029374b6db3df753c1 (DEFAULT_TABLE table=test-table [id=5b94a886c7ab40a385339d86f27013a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:55.142884 16019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc82b5eff640029374b6db3df753c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:55.142985 16153 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc82b5eff640029374b6db3df753c1 (DEFAULT_TABLE table=test-table [id=5b94a886c7ab40a385339d86f27013a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:55.143240 16153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc82b5eff640029374b6db3df753c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:55.145649 16233 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Bootstrap starting.
I20260925 14:51:55.145571 15885 tablet_service.cc:1467] Processing CreateTablet for tablet 67bc82b5eff640029374b6db3df753c1 (DEFAULT_TABLE table=test-table [id=5b94a886c7ab40a385339d86f27013a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:51:55.145900 15885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67bc82b5eff640029374b6db3df753c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:51:55.145952 16234 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Bootstrap starting.
I20260925 14:51:55.146620 16234 tablet_bootstrap.cc:654] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:55.146620 16233 tablet_bootstrap.cc:654] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:55.147230 16233 log.cc:824] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:55.147258 16234 log.cc:824] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:55.148026 16233 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: No bootstrap required, opened a new log
I20260925 14:51:55.148096 16233 ts_tablet_manager.cc:1397] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:51:55.148263 16234 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: No bootstrap required, opened a new log
I20260925 14:51:55.148398 16234 ts_tablet_manager.cc:1397] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:51:55.148964 16237 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap starting.
I20260925 14:51:55.149572 16237 tablet_bootstrap.cc:654] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Neither blocks nor log segments found. Creating new log.
I20260925 14:51:55.150059 16237 log.cc:824] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:51:55.150295 16234 raft_consensus.cc:348] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.150316 16233 raft_consensus.cc:348] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.150508 16234 raft_consensus.cc:374] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:55.150508 16233 raft_consensus.cc:374] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:55.150547 16234 raft_consensus.cc:729] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0fa856dc5347b9b63a2c3de72dfada, State: Initialized, Role: FOLLOWER
I20260925 14:51:55.150547 16233 raft_consensus.cc:729] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ed1117709b4ebe8a9b9714c46ce66d, State: Initialized, Role: FOLLOWER
I20260925 14:51:55.150664 16234 consensus_queue.cc:260] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.150664 16233 consensus_queue.cc:260] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.150934 16234 ts_tablet_manager.cc:1428] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:51:55.150934 16233 ts_tablet_manager.cc:1428] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:55.151008 16237 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: No bootstrap required, opened a new log
I20260925 14:51:55.151053 16219 heartbeater.cc:503] Master 127.0.98.62:35681 was elected leader, sending a full tablet report...
I20260925 14:51:55.151084 16237 ts_tablet_manager.cc:1397] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:51:55.151336 16085 heartbeater.cc:503] Master 127.0.98.62:35681 was elected leader, sending a full tablet report...
I20260925 14:51:55.153141 16237 raft_consensus.cc:348] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.153316 16237 raft_consensus.cc:374] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:51:55.153367 16237 raft_consensus.cc:729] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba6367128f94bbbaebdc4795dbb59c2, State: Initialized, Role: FOLLOWER
I20260925 14:51:55.153487 16237 consensus_queue.cc:260] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.153812 16237 ts_tablet_manager.cc:1428] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:51:55.153854 15951 heartbeater.cc:503] Master 127.0.98.62:35681 was elected leader, sending a full tablet report...
I20260925 14:51:55.201853 16240 raft_consensus.cc:482] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:51:55.201992 16240 raft_consensus.cc:504] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.202431 16240 leader_election.cc:288] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995), dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659)
W20260925 14:51:55.203755 16086 tablet.cc:2369] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:51:55.205554 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.202650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44601 (local address 127.0.98.1:40659)
0925 14:51:55.202698 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.202700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.202712 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:51:55.202749 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.202750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.202763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.202839 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:55.203010 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.203831 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.204804 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.205190 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.205283 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.205294 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.205438 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.205479 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:51:55.205576 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.202683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33349 (local address 127.0.98.2:45995)
0925 14:51:55.202726 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.202728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.202902 (+ 174us) server_negotiation.cc:407] Connection header received
0925 14:51:55.202942 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.202943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.202954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.203028 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:51:55.203182 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.203871 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.204845 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.205213 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.205291 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.205302 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.205476 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.205516 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:51:55.205587 16221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.202587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.98.3:44601)
0925 14:51:55.202640 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.202652 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:55.202697 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:55.202856 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:55.202857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:55.202859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:55.202935 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.202937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.203931 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.203932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.204704 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.204706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.205274 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.205275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.205361 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.205365 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:51:55.205432 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:51:55.205472 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:51:55.205595 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.202630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45995 (local address 127.0.98.3:33349)
0925 14:51:55.202794 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.202826 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:51:55.202880 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:51:55.203046 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:51:55.203047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:51:55.203048 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:51:55.203108 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.203109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.203964 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.203965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.204739 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:51:55.204741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.205304 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.205305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:51:55.205370 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.205373 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:51:55.205457 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:51:55.205497 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":119,"threads_started":1}
I20260925 14:51:55.205996 15905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bc82b5eff640029374b6db3df753c1" candidate_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba6367128f94bbbaebdc4795dbb59c2" is_pre_election: true
I20260925 14:51:55.206060 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bc82b5eff640029374b6db3df753c1" candidate_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" is_pre_election: true
I20260925 14:51:55.206162 15905 raft_consensus.cc:2454] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0fa856dc5347b9b63a2c3de72dfada in term 0.
I20260925 14:51:55.206215 16039 raft_consensus.cc:2454] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7a0fa856dc5347b9b63a2c3de72dfada in term 0.
I20260925 14:51:55.206379 16109 leader_election.cc:302] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0fa856dc5347b9b63a2c3de72dfada, dba6367128f94bbbaebdc4795dbb59c2; no voters:
I20260925 14:51:55.206547 16240 raft_consensus.cc:2781] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:51:55.206636 16240 raft_consensus.cc:482] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:51:55.206662 16240 raft_consensus.cc:3037] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:55.207376 16240 raft_consensus.cc:504] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.207515 16240 leader_election.cc:288] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [CANDIDATE]: Term 1 election: Requested vote from peers b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995), dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659)
I20260925 14:51:55.207669 16039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bc82b5eff640029374b6db3df753c1" candidate_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
I20260925 14:51:55.207683 15905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67bc82b5eff640029374b6db3df753c1" candidate_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dba6367128f94bbbaebdc4795dbb59c2"
I20260925 14:51:55.207755 16039 raft_consensus.cc:3037] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:55.207755 15905 raft_consensus.cc:3037] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:51:55.208631 15905 raft_consensus.cc:2454] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0fa856dc5347b9b63a2c3de72dfada in term 1.
I20260925 14:51:55.208631 16039 raft_consensus.cc:2454] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7a0fa856dc5347b9b63a2c3de72dfada in term 1.
I20260925 14:51:55.208832 16107 leader_election.cc:302] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0fa856dc5347b9b63a2c3de72dfada, b7ed1117709b4ebe8a9b9714c46ce66d; no voters:
I20260925 14:51:55.208940 16240 raft_consensus.cc:2781] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:51:55.209111 16240 raft_consensus.cc:686] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 LEADER]: Becoming Leader. State: Replica: 7a0fa856dc5347b9b63a2c3de72dfada, State: Running, Role: LEADER
I20260925 14:51:55.209183 16240 consensus_queue.cc:237] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:51:55.209955 15759 catalog_manager.cc:5495] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada reported cstate change: term changed from 0 to 1, leader changed from <none> to 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } health_report { overall_health: UNKNOWN } } }
I20260925 14:51:55.233516 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.231387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.0.98.3:44421)
0925 14:51:55.231456 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:51:55.231457 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:51:55.231465 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:51:55.231499 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:51:55.231500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:51:55.231517 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:51:55.231591 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:51:55.231718 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.232179 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.232876 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:51:55.233055 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:51:55.233112 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:51:55.233246 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:51:55.233349 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:51:55.233409 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:51:55.233443 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:51:55.233459 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:51:55.235574 15905 raft_consensus.cc:1261] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Refusing update from remote peer 7a0fa856dc5347b9b63a2c3de72dfada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:55.235579 16039 raft_consensus.cc:1261] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Refusing update from remote peer 7a0fa856dc5347b9b63a2c3de72dfada: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:51:55.235922 16243 consensus_queue.cc:1035] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:55.236050 16240 consensus_queue.cc:1035] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [LEADER]: Connected to new peer: Peer: permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:51:55.240007 16248 mvcc.cc:205] Tried to move back new op lower bound from 7333265060801818624 to 7333265060697432064. Current Snapshot: MvccSnapshot[applied={T|T < 7333265060801818624 or (T in {7333265060801818624})}]
I20260925 14:51:55.240140 16247 mvcc.cc:205] Tried to move back new op lower bound from 7333265060801818624 to 7333265060697432064. Current Snapshot: MvccSnapshot[applied={T|T < 7333265060801818624}]
I20260925 14:51:55.240963 15760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:55.241080 15760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:51:55.241827 15760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Sending DeleteTablet for 3 replicas of tablet 67bc82b5eff640029374b6db3df753c1
I20260925 14:51:55.242141 16153 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc82b5eff640029374b6db3df753c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40032
I20260925 14:51:55.242166 16019 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc82b5eff640029374b6db3df753c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43408
I20260925 14:51:55.242434 16255 tablet_replica.cc:330] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: stopping tablet replica
I20260925 14:51:55.242498 15885 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc82b5eff640029374b6db3df753c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52552
I20260925 14:51:55.242537 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 15822
I20260925 14:51:55.242635 16255 raft_consensus.cc:2229] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:51:55.242784 16255 raft_consensus.cc:2258] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:55.242995 16256 tablet_replica.cc:330] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: stopping tablet replica
I20260925 14:51:55.243120 16256 raft_consensus.cc:2229] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:51:55.243247 16256 raft_consensus.cc:2258] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:51:55.243738 16256 ts_tablet_manager.cc:1905] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:55.244238 16255 ts_tablet_manager.cc:1905] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:51:55.246555 16255 ts_tablet_manager.cc:1918] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:51:55.246694 16255 log.cc:1196] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/67bc82b5eff640029374b6db3df753c1
I20260925 14:51:55.247056 16255 ts_tablet_manager.cc:1939] T 67bc82b5eff640029374b6db3df753c1 P b7ed1117709b4ebe8a9b9714c46ce66d: Deleting consensus metadata
I20260925 14:51:55.247586 15745 catalog_manager.cc:4841] TS b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): tablet 67bc82b5eff640029374b6db3df753c1 (table test-table [id=5b94a886c7ab40a385339d86f27013a0]) successfully deleted
I20260925 14:51:55.248508 16256 ts_tablet_manager.cc:1918] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260925 14:51:55.248513 15747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40659 (error 108)
I20260925 14:51:55.248664 16256 log.cc:1196] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/67bc82b5eff640029374b6db3df753c1
I20260925 14:51:55.248772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 15956
I20260925 14:51:55.249784 16256 ts_tablet_manager.cc:1939] T 67bc82b5eff640029374b6db3df753c1 P 7a0fa856dc5347b9b63a2c3de72dfada: Deleting consensus metadata
I20260925 14:51:55.250267 15744 catalog_manager.cc:4841] TS 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): tablet 67bc82b5eff640029374b6db3df753c1 (table test-table [id=5b94a886c7ab40a385339d86f27013a0]) successfully deleted
I20260925 14:51:55.251103 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:51:55.250979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.0.1:52562)
0925 14:51:55.251036 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:51:55.251077 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:51:55.254519 15747 catalog_manager.cc:4584] TS dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67bc82b5eff640029374b6db3df753c1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40659: connect: Connection refused (error 111)
I20260925 14:51:55.255169 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 16090
I20260925 14:51:55.260316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 15728
I20260925 14:52:15.265112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--webserver_interface=127.0.98.62
--webserver_port=44625
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:15.344748 16258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:15.344954 16258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:15.344976 16258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:15.346132 16258 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:52:15.346174 16258 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:52:15.346187 16258 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:52:15.346199 16258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:15.346211 16258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:15.346225 16258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:15.346237 16258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:15.346304 16258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:15.347791 16258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:15.348034 16258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:15.348292 16258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:15.357645 16264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.357612 16265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.357604 16267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:15.357657 16258 server_base.cc:1034] running on GCE node
I20260925 14:52:15.358232 16258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:15.358632 16258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:15.359820 16258 hybrid_clock.cc:648] HybridClock initialized: now 1790347935359802 us; error 30 us; skew 500 ppm
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:15.367740 16258 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:15.368299 16258 webserver.cc:460] Webserver started at http://127.0.98.62:44625/ using document root <none> and password file <none>
I20260925 14:52:15.368551 16258 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:15.368637 16258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:15.370232 16258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:52:15.370994 16274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.371162 16258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.371273 16258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "4c0467d46758467fb61de3ee87367a2e"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "cad28d15f3428b70a145576bb279ea18"
server_key_iv: "d2999a81721005667fbce5318604355c"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.371688 16258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:15.392040 16258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:15.394488 16258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:15.394733 16258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:15.399466 16258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35681
I20260925 14:52:15.399500 16326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35681 every 8 connection(s)
I20260925 14:52:15.399960 16258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:15.400575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16258
I20260925 14:52:15.400906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35681
--webserver_port=46031
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:15.402456 16327 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:52:15.403152 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap starting.
I20260925 14:52:15.412773 16327 log.cc:824] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:15.415256 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:15.415611 16327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap complete.
I20260925 14:52:15.417959 16327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:15.418200 16327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0467d46758467fb61de3ee87367a2e, State: Initialized, Role: FOLLOWER
I20260925 14:52:15.418375 16327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:15.418502 16327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:15.418561 16327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:15.418613 16327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:52:15.419672 16327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:15.419795 16327 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0467d46758467fb61de3ee87367a2e; no voters:
I20260925 14:52:15.419991 16327 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:52:15.420027 16331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:52:15.420138 16331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 LEADER]: Becoming Leader. State: Replica: 4c0467d46758467fb61de3ee87367a2e, State: Running, Role: LEADER
I20260925 14:52:15.420269 16327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:15.420230 16331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:15.420521 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0467d46758467fb61de3ee87367a2e. Latest consensus state: current_term: 2 leader_uuid: "4c0467d46758467fb61de3ee87367a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260925 14:52:15.420598 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:15.420527 16332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4c0467d46758467fb61de3ee87367a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260925 14:52:15.420789 16332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:15.420804 16335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:15.421792 16335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:15.421953 16335 catalog_manager.cc:1259] Loaded cluster ID: 9193d852cd57483a8eec21b5431a536a
I20260925 14:52:15.422024 16335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:15.423487 16335 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:15.423678 16335 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Loaded TSK: 0
I20260925 14:52:15.435637 16335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Generated new TSK 1
I20260925 14:52:15.435750 16335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:52:15.493424 16329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:15.493630 16329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:15.493654 16329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:15.494997 16329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:15.495045 16329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:15.495074 16329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:15.496551 16329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46031
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:15.496804 16329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:15.497056 16329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:15.501966 16354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.501964 16357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.501957 16355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:15.502506 16329 server_base.cc:1034] running on GCE node
I20260925 14:52:15.502753 16329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:15.503070 16329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:15.504232 16329 hybrid_clock.cc:648] HybridClock initialized: now 1790347935504212 us; error 32 us; skew 500 ppm
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:15.507073 16329 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:15.507615 16329 webserver.cc:460] Webserver started at http://127.0.98.1:46031/ using document root <none> and password file <none>
I20260925 14:52:15.507875 16329 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:15.507961 16329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:15.509361 16329 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:52:15.510051 16364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.510265 16329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:15.510385 16329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "dba6367128f94bbbaebdc4795dbb59c2"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "7fde3e8e98bcf01812166df18875960b"
server_key_iv: "e1a3e4f94b18a008a714a4f742f20abe"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.510792 16329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:15.532097 16329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:15.534305 16329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:15.534520 16329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:15.534771 16329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:15.535300 16371 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:52:15.536749 16329 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:52:15.536800 16329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:52:15.536835 16329 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:52:15.537510 16329 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:52:15.537554 16329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:15.537601 16371 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap starting.
I20260925 14:52:15.545256 16329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40659
I20260925 14:52:15.545804 16329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:15.546618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16329
I20260925 14:52:15.546957 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45995
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35681
--webserver_port=42613
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:15.549192 16371 log.cc:824] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:15.549539 16478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40659 every 8 connection(s)
I20260925 14:52:15.552431 16371 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:15.552769 16371 tablet_bootstrap.cc:492] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap complete.
I20260925 14:52:15.554054 16371 ts_tablet_manager.cc:1397] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:15.555809 16371 raft_consensus.cc:348] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.556208 16371 raft_consensus.cc:729] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba6367128f94bbbaebdc4795dbb59c2, State: Initialized, Role: FOLLOWER
I20260925 14:52:15.556386 16371 consensus_queue.cc:260] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.556694 16371 ts_tablet_manager.cc:1428] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20260925 14:52:15.560200 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.546605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33041 (local address 127.0.98.62:35681)
0925 14:52:15.549530 (+ 2925us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.549540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.550457 (+ 917us) server_negotiation.cc:407] Connection header received
0925 14:52:15.550595 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.550602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.550658 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.550803 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.551708 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.552670 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.553875 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.554340 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.554493 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.556769 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.556774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.556774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.556786 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.558497 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.558927 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.558930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.558930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.559027 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.559289 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.559290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.559290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.559508 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.559590 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.559891 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.560013 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2867,"thread_start_us":82,"threads_started":1}
I20260925 14:52:15.560200 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.546554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:33041)
0925 14:52:15.549626 (+ 3072us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.549673 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.550270 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.550939 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.550943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.551309 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:15.551571 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.551578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.552793 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.552795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.553768 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.553770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.554489 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.554490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.554627 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.555188 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:15.555196 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:15.556632 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:15.558593 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.558600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.558602 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.558827 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.559102 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.559102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.559103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.559207 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.559584 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:15.559590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:15.559725 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.559839 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.560010 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3030,"thread_start_us":98,"threads_started":1}
I20260925 14:52:15.561287 16479 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260925 14:52:15.561365 16479 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:15.561499 16479 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260925 14:52:15.562131 16291 ts_manager.cc:194] Registered new tserver with Master: dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659)
I20260925 14:52:15.563215 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33041
I20260925 14:52:15.564245 16479 heartbeater.cc:503] Master 127.0.98.62:35681 was elected leader, sending a full tablet report...
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260925 14:52:15.571584 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.562775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.0.1:38228)
0925 14:52:15.563617 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.563647 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.563737 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.563950 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.563956 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.564286 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:15.564342 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.564344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.565614 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.565615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.566228 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.566230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.566869 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.566871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.566927 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.567431 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:15.567436 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:15.568660 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:15.570523 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.570529 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.570531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.570712 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.570925 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.570926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.570926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.570976 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.571288 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:15.571295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:15.571353 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.571419 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.571517 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"thread_start_us":93,"threads_started":1}
I20260925 14:52:15.571640 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.562865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.0.98.1:40659)
0925 14:52:15.563553 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.563558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.563722 (+ 164us) server_negotiation.cc:407] Connection header received
0925 14:52:15.563819 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.563822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.563850 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.563942 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.564423 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.565531 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.566319 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.566797 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.566900 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.568737 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.568744 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.568745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.568752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.570450 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.570789 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.570792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.570792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.570855 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.571034 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.571035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.571035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.571177 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.571254 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.571473 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.571554 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":72,"threads_started":1}
I20260925 14:52:15.571970 16413 tablet_service.cc:1514] Processing DeleteTablet for tablet 67bc82b5eff640029374b6db3df753c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:51:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38228
I20260925 14:52:15.572513 16488 tablet_replica.cc:330] stopping tablet replica
I20260925 14:52:15.572630 16488 raft_consensus.cc:2229] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:15.572713 16488 raft_consensus.cc:2258] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:15.573009 16488 ts_tablet_manager.cc:1905] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:15.575165 16488 ts_tablet_manager.cc:1918] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:15.575243 16488 log.cc:1196] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal/wals/67bc82b5eff640029374b6db3df753c1
I20260925 14:52:15.575448 16488 ts_tablet_manager.cc:1939] T 67bc82b5eff640029374b6db3df753c1 P dba6367128f94bbbaebdc4795dbb59c2: Deleting consensus metadata
I20260925 14:52:15.575960 16278 catalog_manager.cc:4841] TS dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659): tablet 67bc82b5eff640029374b6db3df753c1 (table test-table [id=5b94a886c7ab40a385339d86f27013a0]) successfully deleted
W20260925 14:52:15.632802 16483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:15.633044 16483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:15.633080 16483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:15.634217 16483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:15.634291 16483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:15.634335 16483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:15.635816 16483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42613
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:15.636075 16483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:15.636307 16483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:15.641219 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.641269 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.641255 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:15.641304 16483 server_base.cc:1034] running on GCE node
I20260925 14:52:15.641572 16483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:15.641816 16483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:15.642956 16483 hybrid_clock.cc:648] HybridClock initialized: now 1790347935642946 us; error 27 us; skew 500 ppm
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:15.645435 16483 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:15.645861 16483 webserver.cc:460] Webserver started at http://127.0.98.2:42613/ using document root <none> and password file <none>
I20260925 14:52:15.646093 16483 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:15.646152 16483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:15.647542 16483 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:15.648154 16503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.648329 16483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:15.648399 16483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "543a507efe568b01b86bd92e406f3d25"
server_key_iv: "2e6c0186d42d850657fe13f874d82b28"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.648741 16483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:15.657104 16483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:15.658987 16483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:15.659138 16483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:15.659359 16483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:15.659660 16483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:15.659715 16483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.659746 16483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:15.659773 16483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.665756 16483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45995
I20260925 14:52:15.665807 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45995 every 8 connection(s)
I20260925 14:52:15.666173 16483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:15.672897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16483
I20260925 14:52:15.673223 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44421
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35681
--webserver_port=40065
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:15.676527 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.666956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60389 (local address 127.0.98.62:35681)
0925 14:52:15.667001 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.667003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.667191 (+ 188us) server_negotiation.cc:407] Connection header received
0925 14:52:15.667721 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.667722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.667738 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.667792 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.669048 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.669553 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.670648 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.670866 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.670972 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.673222 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.673223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.673224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.673232 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.674908 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.675424 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.675425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.675425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.675482 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.675772 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.675772 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.675772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.675899 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.675950 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.676355 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.676456 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:52:15.676779 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.666921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:60389)
0925 14:52:15.667063 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.667106 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.667613 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.667869 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.667873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.668437 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:15.668913 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.668919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.669650 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.669651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.670558 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.670561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.670990 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.670991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.671120 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.671664 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:15.671674 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:15.673129 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:15.674989 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.674996 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.674998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.675326 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.675558 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.675560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.675560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.675702 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.675968 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:15.675974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:15.676153 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.676288 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.676495 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260925 14:52:15.677347 16617 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260925 14:52:15.677462 16617 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:15.677685 16617 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260925 14:52:15.678172 16291 ts_manager.cc:194] Registered new tserver with Master: b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
I20260925 14:52:15.678596 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60389
W20260925 14:52:15.755893 16620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:15.756088 16620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:15.756111 16620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:15.757256 16620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:15.757303 16620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:15.757330 16620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:15.758828 16620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40065
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:15.759065 16620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:15.759303 16620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:15.764312 16625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.764302 16628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:15.764328 16626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:15.764613 16620 server_base.cc:1034] running on GCE node
I20260925 14:52:15.764837 16620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:15.765076 16620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:15.766239 16620 hybrid_clock.cc:648] HybridClock initialized: now 1790347935766231 us; error 48 us; skew 500 ppm
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:15.768911 16620 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:15.769311 16620 webserver.cc:460] Webserver started at http://127.0.98.3:40065/ using document root <none> and password file <none>
I20260925 14:52:15.769519 16620 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:15.769570 16620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:15.770951 16620 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:52:15.771643 16635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.771991 16620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:15.772095 16620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "7a0fa856dc5347b9b63a2c3de72dfada"
format_stamp: "Formatted at 2026-09-25 14:51:55 on dist-test-slave-z2s5"
server_key: "a6855c98fa76585470b989555d6eec2a"
server_key_iv: "e13f1ff88a26f64af8dfe4848f79dc8e"
server_key_version: "encryptionkey@0"
I20260925 14:52:15.772439 16620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:15.782825 16620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:15.785147 16620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:15.785305 16620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:15.785530 16620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:15.785837 16620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:15.785873 16620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.785914 16620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:15.785940 16620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:15.791986 16620 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44421
I20260925 14:52:15.792032 16748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44421 every 8 connection(s)
I20260925 14:52:15.792372 16620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:52:15 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:15.799144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16620
I20260925 14:52:15.802176 16482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.793045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41735 (local address 127.0.98.62:35681)
0925 14:52:15.793115 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.793117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.793334 (+ 217us) server_negotiation.cc:407] Connection header received
0925 14:52:15.793811 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.793812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.793821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.793891 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.794926 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.795413 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.796530 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.796751 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.796836 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.799104 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.799105 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.799105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.799112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.800678 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.801100 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.801101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.801101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.801168 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.801444 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.801445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.801445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.801549 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.801601 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.802030 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.802107 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:52:15.802351 16751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.793032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:41735)
0925 14:52:15.793205 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.793248 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.793722 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.793943 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.793947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.794603 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:15.794806 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.794812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.795529 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.795530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.796415 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.796418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.796842 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.796843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.796951 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.797466 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:15.797474 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:15.799002 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:15.800760 (+ 1758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.800807 (+ 47us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.800809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.801027 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.801235 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:15.801236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:15.801236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:15.801366 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:15.801613 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:15.801618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:15.801837 (+ 219us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.801956 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.802117 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":106,"threads_started":1}
I20260925 14:52:15.802966 16749 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35681
I20260925 14:52:15.803066 16749 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:15.803287 16749 heartbeater.cc:511] Master 127.0.98.62:35681 requested a full tablet report, sending...
I20260925 14:52:15.803737 16291 ts_manager.cc:194] Registered new tserver with Master: 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421)
I20260925 14:52:15.804275 16291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41735
I20260925 14:52:15.807065 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.799590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.0.98.62:35681)
0925 14:52:15.799747 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.799749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.799817 (+ 68us) server_negotiation.cc:407] Connection header received
0925 14:52:15.799920 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.799921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.799943 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.800011 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:15.800642 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.801299 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.802334 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.802767 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.802871 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.804439 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:15.804440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:15.804440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:15.804447 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:15.805851 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.806239 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.806240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.806240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.806299 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:15.806554 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:15.806555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:15.806555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:15.806657 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:15.806782 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.806866 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.806935 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":103,"threads_started":1}
I20260925 14:52:15.807801 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:15.810874 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.808168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.0.98.62:35681)
0925 14:52:15.808225 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.808227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.808328 (+ 101us) server_negotiation.cc:407] Connection header received
0925 14:52:15.808398 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.808399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.808423 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.808479 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:15.808598 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.809061 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.809806 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.810197 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.810264 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.810455 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:15.810554 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:15.810621 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.810805 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.810829 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:52:15.811201 16291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:15.812916 16291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:15.815522 16413 tablet_service.cc:1467] Processing CreateTablet for tablet ead08fac40a74c67999d20e4742c42f9 (DEFAULT_TABLE table=test-table [id=e600e7f0f5ad45a3aaefed78bb15599e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:15.815764 16413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead08fac40a74c67999d20e4742c42f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:15.817291 16371 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap starting.
I20260925 14:52:15.817703 16371 tablet_bootstrap.cc:654] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:15.818584 16371 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: No bootstrap required, opened a new log
I20260925 14:52:15.818637 16371 ts_tablet_manager.cc:1397] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:15.819021 16371 raft_consensus.cc:348] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.819100 16371 raft_consensus.cc:374] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:15.819150 16371 raft_consensus.cc:729] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba6367128f94bbbaebdc4795dbb59c2, State: Initialized, Role: FOLLOWER
I20260925 14:52:15.819221 16371 consensus_queue.cc:260] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.819339 16371 ts_tablet_manager.cc:1428] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:15.819737 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.815284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45995 (local address 127.0.0.1:59944)
0925 14:52:15.815444 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.815455 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.815497 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.815944 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.815945 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.815947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:15.815997 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.815999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.816934 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.816935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.818230 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.818233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.819467 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.819469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.819540 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.819562 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.819618 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.819662 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260925 14:52:15.820053 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.815539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.0.98.2:45995)
0925 14:52:15.815763 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.815769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.815780 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:52:15.815823 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.815826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.815843 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.815932 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:15.816078 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.816830 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.818331 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.819274 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.819353 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.819424 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.819917 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.819966 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
I20260925 14:52:15.820916 16551 tablet_service.cc:1467] Processing CreateTablet for tablet ead08fac40a74c67999d20e4742c42f9 (DEFAULT_TABLE table=test-table [id=e600e7f0f5ad45a3aaefed78bb15599e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:15.821326 16551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead08fac40a74c67999d20e4742c42f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:15.821692 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.815200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44421 (local address 127.0.0.1:42610)
0925 14:52:15.815239 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.815253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.815307 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.817604 (+ 2297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.817605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.817608 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:15.817644 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.817645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.818886 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.818887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.819873 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.819877 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.821374 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.821376 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.821457 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.821492 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.821564 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.821618 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:52:15.821730 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.815347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42610 (local address 127.0.98.3:44421)
0925 14:52:15.817397 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.817402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.817415 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:52:15.817467 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.817470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.817491 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.817602 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:15.817708 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.818771 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.820503 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.821280 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.821372 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.821435 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.821582 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.821627 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2007,"thread_start_us":97,"threads_started":1}
I20260925 14:52:15.822463 16683 tablet_service.cc:1467] Processing CreateTablet for tablet ead08fac40a74c67999d20e4742c42f9 (DEFAULT_TABLE table=test-table [id=e600e7f0f5ad45a3aaefed78bb15599e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:15.822811 16683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead08fac40a74c67999d20e4742c42f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:15.824934 16759 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Bootstrap starting.
I20260925 14:52:15.825580 16759 tablet_bootstrap.cc:654] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:15.826052 16760 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Bootstrap starting.
I20260925 14:52:15.826172 16759 log.cc:824] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:15.826807 16760 tablet_bootstrap.cc:654] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:15.827111 16759 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: No bootstrap required, opened a new log
I20260925 14:52:15.827243 16759 ts_tablet_manager.cc:1397] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:15.827534 16760 log.cc:824] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:15.828332 16760 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: No bootstrap required, opened a new log
I20260925 14:52:15.828428 16760 ts_tablet_manager.cc:1397] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:15.828884 16759 raft_consensus.cc:348] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.829046 16759 raft_consensus.cc:374] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:15.829070 16759 raft_consensus.cc:729] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ed1117709b4ebe8a9b9714c46ce66d, State: Initialized, Role: FOLLOWER
I20260925 14:52:15.829203 16759 consensus_queue.cc:260] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.829520 16759 ts_tablet_manager.cc:1428] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:15.829566 16617 heartbeater.cc:503] Master 127.0.98.62:35681 was elected leader, sending a full tablet report...
I20260925 14:52:15.830510 16760 raft_consensus.cc:348] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.830730 16760 raft_consensus.cc:374] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:15.830777 16760 raft_consensus.cc:729] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0fa856dc5347b9b63a2c3de72dfada, State: Initialized, Role: FOLLOWER
I20260925 14:52:15.830914 16760 consensus_queue.cc:260] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.831238 16760 ts_tablet_manager.cc:1428] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:15.831290 16749 heartbeater.cc:503] Master 127.0.98.62:35681 was elected leader, sending a full tablet report...
W20260925 14:52:15.916775 16618 tablet.cc:2369] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:15.969806 16485 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:15.970003 16485 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.970381 16485 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
I20260925 14:52:15.973356 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.970612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45995 (local address 127.0.98.1:33749)
0925 14:52:15.970678 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.970692 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.970773 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.970973 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.970974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.970975 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:15.971057 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.971059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.971772 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.971773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.972523 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.972525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.973071 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.973072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.973157 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.973173 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.973246 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.973286 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:52:15.973354 16757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.970798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33749 (local address 127.0.98.2:45995)
0925 14:52:15.970848 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.970850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.970857 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:52:15.970892 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.970892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.970902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.970969 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:15.971146 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.971630 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.972616 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.972992 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.973073 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.973083 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.973251 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.973287 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:52:15.973752 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.970829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57251 (local address 127.0.98.3:44421)
0925 14:52:15.971019 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.971021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.971028 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:52:15.971051 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.971051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.971061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.971135 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:15.971339 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.971846 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.972948 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.973368 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.973458 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.973467 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:15.973657 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:52:15.973708 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260925 14:52:15.973707 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead08fac40a74c67999d20e4742c42f9" candidate_uuid: "dba6367128f94bbbaebdc4795dbb59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" is_pre_election: true
I20260925 14:52:15.973791 16765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.970635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44421 (local address 127.0.98.1:57251)
0925 14:52:15.970875 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:15.970888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:15.970946 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:15.971139 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:15.971140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:15.971142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:15.971237 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.971239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.971954 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.971956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.972840 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:15.972841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:15.973458 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:15.973459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:15.973541 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:15.973545 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:15.973640 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:52:15.973679 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":120,"threads_started":1}
I20260925 14:52:15.973886 16571 raft_consensus.cc:2454] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba6367128f94bbbaebdc4795dbb59c2 in term 0.
I20260925 14:52:15.974164 16366 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7ed1117709b4ebe8a9b9714c46ce66d, dba6367128f94bbbaebdc4795dbb59c2; no voters:
I20260925 14:52:15.974161 16703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead08fac40a74c67999d20e4742c42f9" candidate_uuid: "dba6367128f94bbbaebdc4795dbb59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" is_pre_election: true
I20260925 14:52:15.974303 16703 raft_consensus.cc:2454] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dba6367128f94bbbaebdc4795dbb59c2 in term 0.
I20260925 14:52:15.974362 16485 raft_consensus.cc:2781] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:15.974467 16485 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:15.974500 16485 raft_consensus.cc:3037] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:15.975276 16485 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.975450 16485 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 1 election: Requested vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
I20260925 14:52:15.975620 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead08fac40a74c67999d20e4742c42f9" candidate_uuid: "dba6367128f94bbbaebdc4795dbb59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
I20260925 14:52:15.975589 16703 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead08fac40a74c67999d20e4742c42f9" candidate_uuid: "dba6367128f94bbbaebdc4795dbb59c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a0fa856dc5347b9b63a2c3de72dfada"
I20260925 14:52:15.975690 16571 raft_consensus.cc:3037] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:15.975690 16703 raft_consensus.cc:3037] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:15.976557 16703 raft_consensus.cc:2454] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba6367128f94bbbaebdc4795dbb59c2 in term 1.
I20260925 14:52:15.976557 16571 raft_consensus.cc:2454] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dba6367128f94bbbaebdc4795dbb59c2 in term 1.
I20260925 14:52:15.976790 16365 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a0fa856dc5347b9b63a2c3de72dfada, dba6367128f94bbbaebdc4795dbb59c2; no voters:
I20260925 14:52:15.976895 16485 raft_consensus.cc:2781] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:15.977033 16485 raft_consensus.cc:686] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 LEADER]: Becoming Leader. State: Replica: dba6367128f94bbbaebdc4795dbb59c2, State: Running, Role: LEADER
I20260925 14:52:15.977116 16485 consensus_queue.cc:237] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:15.977921 16291 catalog_manager.cc:5495] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dba6367128f94bbbaebdc4795dbb59c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:16.001600 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:15.999216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.0.98.1:40659)
0925 14:52:15.999289 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:15.999291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:15.999301 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:15.999348 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:15.999349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:15.999366 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:15.999442 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:15.999574 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.000081 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.000849 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.001069 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.001150 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.001305 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:16.001355 (+ 50us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 14:52:16.001511 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38236: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260925 14:52:16.001688 16487 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38236: token expired
I20260925 14:52:16.007937 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.001963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.0.98.62:35681)
0925 14:52:16.002025 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.002027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.002037 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:16.002086 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.002086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.002094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.002140 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:16.002678 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.003165 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.003852 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.004031 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.004084 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.005243 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:16.005244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:16.005244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:16.005250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:16.006854 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.007196 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.007197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.007197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.007252 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:16.007425 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:16.007436 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:16.007437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:16.007536 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:16.007618 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.007807 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.007872 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:52:16.011718 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.008985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.98.1:40659)
0925 14:52:16.009061 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:16.009064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:16.009075 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:52:16.009103 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:16.009104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:16.009116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:16.009207 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:16.009359 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.009879 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.010818 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:16.011031 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:16.011139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:16.011310 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:16.011423 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:16.011514 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:16.011589 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:52:16.011611 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:52:16.013459 16571 raft_consensus.cc:1261] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Refusing update from remote peer dba6367128f94bbbaebdc4795dbb59c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:16.013501 16703 raft_consensus.cc:1261] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Refusing update from remote peer dba6367128f94bbbaebdc4795dbb59c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:16.013787 16766 consensus_queue.cc:1035] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:16.013970 16766 consensus_queue.cc:1035] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:16.017035 16771 mvcc.cc:205] Tried to move back new op lower bound from 7333265145908105216 to 7333265145762836480. Current Snapshot: MvccSnapshot[applied={T|T < 7333265145908105216}]
I20260925 14:52:16.017079 16770 mvcc.cc:205] Tried to move back new op lower bound from 7333265145908105216 to 7333265145762836480. Current Snapshot: MvccSnapshot[applied={T|T < 7333265145908105216}]
I20260925 14:52:16.018486 16291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:16.018582 16291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:16.019289 16291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Sending DeleteTablet for 3 replicas of tablet ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:16.019613 16413 tablet_service.cc:1514] Processing DeleteTablet for tablet ead08fac40a74c67999d20e4742c42f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38228
I20260925 14:52:16.019640 16683 tablet_service.cc:1514] Processing DeleteTablet for tablet ead08fac40a74c67999d20e4742c42f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42610
I20260925 14:52:16.019723 16488 tablet_replica.cc:330] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: stopping tablet replica
I20260925 14:52:16.019670 16551 tablet_service.cc:1514] Processing DeleteTablet for tablet ead08fac40a74c67999d20e4742c42f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59944
I20260925 14:52:16.019775 16488 raft_consensus.cc:2229] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:16.019907 16780 tablet_replica.cc:330] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: stopping tablet replica
I20260925 14:52:16.019907 16488 raft_consensus.cc:2258] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:16.019997 16780 raft_consensus.cc:2229] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:16.020015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 16329
I20260925 14:52:16.020089 16780 raft_consensus.cc:2258] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:16.021118 16780 ts_tablet_manager.cc:1905] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:16.022944 16780 ts_tablet_manager.cc:1918] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:16.023039 16780 log.cc:1196] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:16.023305 16780 ts_tablet_manager.cc:1939] T ead08fac40a74c67999d20e4742c42f9 P b7ed1117709b4ebe8a9b9714c46ce66d: Deleting consensus metadata
I20260925 14:52:16.023774 16276 catalog_manager.cc:4841] TS b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): tablet ead08fac40a74c67999d20e4742c42f9 (table test-table [id=e600e7f0f5ad45a3aaefed78bb15599e]) successfully deleted
W20260925 14:52:16.026610 16278 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40659 (error 108)
I20260925 14:52:16.026779 16779 tablet_replica.cc:330] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: stopping tablet replica
I20260925 14:52:16.026957 16779 raft_consensus.cc:2229] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:16.027009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 16483
I20260925 14:52:16.027094 16779 raft_consensus.cc:2258] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:16.027225 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:16.027097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40659 (local address 127.0.0.1:38240)
0925 14:52:16.027150 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:16.027192 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:52:16.027663 16779 ts_tablet_manager.cc:1905] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:52:16.027370 16278 catalog_manager.cc:4584] TS dba6367128f94bbbaebdc4795dbb59c2 (127.0.98.1:40659): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ead08fac40a74c67999d20e4742c42f9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40659: connect: Connection refused (error 111)
I20260925 14:52:16.029114 16779 ts_tablet_manager.cc:1918] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:16.029176 16779 log.cc:1196] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:16.029479 16779 ts_tablet_manager.cc:1939] T ead08fac40a74c67999d20e4742c42f9 P 7a0fa856dc5347b9b63a2c3de72dfada: Deleting consensus metadata
I20260925 14:52:16.030045 16275 catalog_manager.cc:4841] TS 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): tablet ead08fac40a74c67999d20e4742c42f9 (table test-table [id=e600e7f0f5ad45a3aaefed78bb15599e]) successfully deleted
I20260925 14:52:16.033591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 16620
I20260925 14:52:16.038043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 16258
I20260925 14:52:36.044281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--webserver_interface=127.0.98.62
--webserver_port=44625
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:36.125762 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:36.125999 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:36.126035 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:36.127249 16782 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 14:52:36.127312 16782 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 14:52:36.127334 16782 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 14:52:36.127355 16782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:36.127386 16782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:36.127413 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:36.127435 16782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:36.127462 16782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:36.128938 16782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:36.129213 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:36.129451 16782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:36.134833 16790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.134877 16788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.134855 16782 server_base.cc:1034] running on GCE node
W20260925 14:52:36.134856 16787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.135263 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:36.135564 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:36.136732 16782 hybrid_clock.cc:648] HybridClock initialized: now 1790347956136698 us; error 47 us; skew 500 ppm
Sep 25 14:52:36 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:36.147796 16782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:36.148375 16782 webserver.cc:460] Webserver started at http://127.0.98.62:44625/ using document root <none> and password file <none>
I20260925 14:52:36.148640 16782 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:36.148695 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:36.150224 16782 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:36.151077 16797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.151270 16782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:36.151374 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "4c0467d46758467fb61de3ee87367a2e"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "cad28d15f3428b70a145576bb279ea18"
server_key_iv: "d2999a81721005667fbce5318604355c"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.151762 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:36.163816 16782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:36.166307 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:36.166500 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:36.170634 16782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35681
I20260925 14:52:36.170712 16849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35681 every 8 connection(s)
I20260925 14:52:36.171100 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:36.172652 16850 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:52:36.173333 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap starting.
I20260925 14:52:36.179603 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16782
I20260925 14:52:36.179950 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40659
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35681
--webserver_port=46031
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:36.182051 16850 log.cc:824] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:36.184358 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:36.184682 16850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Bootstrap complete.
I20260925 14:52:36.187026 16850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:36.187281 16850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0467d46758467fb61de3ee87367a2e, State: Initialized, Role: FOLLOWER
I20260925 14:52:36.187465 16850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:36.187548 16850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:36.187628 16850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:36.187680 16850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:52:36.188697 16850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:36.188822 16850 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0467d46758467fb61de3ee87367a2e; no voters:
I20260925 14:52:36.189039 16850 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 14:52:36.189074 16854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:52:36.189195 16854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [term 3 LEADER]: Becoming Leader. State: Replica: 4c0467d46758467fb61de3ee87367a2e, State: Running, Role: LEADER
I20260925 14:52:36.189282 16854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } }
I20260925 14:52:36.189309 16850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:36.189885 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4c0467d46758467fb61de3ee87367a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260925 14:52:36.189901 16855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0467d46758467fb61de3ee87367a2e. Latest consensus state: current_term: 3 leader_uuid: "4c0467d46758467fb61de3ee87367a2e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0467d46758467fb61de3ee87367a2e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35681 } } }
I20260925 14:52:36.190106 16855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:36.190318 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:36.190317 16863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:36.191318 16863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:36.191624 16863 catalog_manager.cc:1259] Loaded cluster ID: 9193d852cd57483a8eec21b5431a536a
I20260925 14:52:36.191663 16863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:36.192859 16863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:36.193032 16863 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Loaded TSK: 0
I20260925 14:52:36.193075 16863 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4c0467d46758467fb61de3ee87367a2e: Loaded TSK: 1
F20260925 14:52:36.193338 16863 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff1ee0549ad google::LogMessage::Fail() at ??:0
@ 0x7ff1ee05788a google::LogMessage::SendToLog() at ??:0
@ 0x7ff1ee0544de google::LogMessage::Flush() at ??:0
@ 0x7ff1ee055a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff1f07c1104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff1f07bed41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff1f5fae1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff1f5fae48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff1f5fae594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff1f5f74c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff1f5f7ead0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff1eff220d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff1eff17954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff1f0d7b6db start_thread at ??:0
@ 0x7ff1ecb5261f clone at ??:0
W20260925 14:52:36.286614 16852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:36.286866 16852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:36.286907 16852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:36.288211 16852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:36.288295 16852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:36.288345 16852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:36.289954 16852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46031
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:36.290253 16852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:36.290565 16852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:36.296011 16892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.296046 16894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.296082 16891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.296487 16852 server_base.cc:1034] running on GCE node
I20260925 14:52:36.296726 16852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:36.297006 16852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:36.298182 16852 hybrid_clock.cc:648] HybridClock initialized: now 1790347956298161 us; error 36 us; skew 500 ppm
Sep 25 14:52:36 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:36.301041 16852 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:36.301527 16852 webserver.cc:460] Webserver started at http://127.0.98.1:46031/ using document root <none> and password file <none>
I20260925 14:52:36.301779 16852 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:36.301855 16852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:36.303462 16852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:36.304268 16901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.304487 16852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:36.304600 16852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "dba6367128f94bbbaebdc4795dbb59c2"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "7fde3e8e98bcf01812166df18875960b"
server_key_iv: "e1a3e4f94b18a008a714a4f742f20abe"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.305053 16852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:36.316327 16852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:36.318933 16852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:36.319082 16852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:36.319360 16852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:36.319870 16908 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 14:52:36.321327 16852 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 14:52:36.321384 16852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 14:52:36.321410 16852 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 14:52:36.322104 16852 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 14:52:36.322149 16852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:36.322249 16908 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap starting.
I20260925 14:52:36.329972 16852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40659
I20260925 14:52:36.330139 17015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40659 every 8 connection(s)
I20260925 14:52:36.330541 16852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:36.334235 16908 log.cc:824] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:36.335445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 16852
I20260925 14:52:36.335780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45995
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35681
--webserver_port=42613
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:36.336786 16908 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:52:36.337201 16908 tablet_bootstrap.cc:492] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Bootstrap complete.
I20260925 14:52:36.338464 16908 ts_tablet_manager.cc:1397] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.013s
I20260925 14:52:36.340554 16908 raft_consensus.cc:348] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:36.340962 16908 raft_consensus.cc:729] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dba6367128f94bbbaebdc4795dbb59c2, State: Initialized, Role: FOLLOWER
I20260925 14:52:36.341157 16908 consensus_queue.cc:260] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:36.341509 16908 ts_tablet_manager.cc:1428] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
W20260925 14:52:36.405107 17018 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:52:36.331276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:51179)
0925 14:52:36.331465 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.331510 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:36.332321 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:36.404892 (+ 72571us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
W20260925 14:52:36.405470 17016 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35681 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35681: Client connection negotiation failed: client connection to 127.0.98.62:35681: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 14:52:36.405864 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.405740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:44227)
0925 14:52:36.405797 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.405841 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:52:36.406384 17018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.406279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:49929)
0925 14:52:36.406339 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.406366 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 14:52:36.406599 17016 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:52:36.427242 17020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:36.427496 17020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:36.427541 17020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:36.428756 17020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:36.428838 17020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:36.428889 17020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:36.430478 17020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42613
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:36.430749 17020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:36.430992 17020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:36.436103 17029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.436098 17027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.436098 17026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.436414 17020 server_base.cc:1034] running on GCE node
I20260925 14:52:36.436618 17020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:36.436874 17020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:36.438031 17020 hybrid_clock.cc:648] HybridClock initialized: now 1790347956438021 us; error 29 us; skew 500 ppm
Sep 25 14:52:36 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:36.440622 17020 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:36.441103 17020 webserver.cc:460] Webserver started at http://127.0.98.2:42613/ using document root <none> and password file <none>
I20260925 14:52:36.441357 17020 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:36.441429 17020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:36.442862 17020 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:52:36.443564 17036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.443780 17020 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:52:36.443886 17020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "b7ed1117709b4ebe8a9b9714c46ce66d"
format_stamp: "Formatted at 2026-09-25 14:51:54 on dist-test-slave-z2s5"
server_key: "543a507efe568b01b86bd92e406f3d25"
server_key_iv: "2e6c0186d42d850657fe13f874d82b28"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.444340 17020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:36.452214 17020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:36.454186 17020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:36.454347 17020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:36.454600 17020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:36.454907 17020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:36.454962 17020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.455001 17020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:36.455029 17020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.461022 17020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45995
I20260925 14:52:36.461064 17149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45995 every 8 connection(s)
I20260925 14:52:36.461413 17020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:36.462088 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17020
I20260925 14:52:36.462363 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44421
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35681
--webserver_port=40065
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:52:36.462750 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.462105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:58331)
0925 14:52:36.462339 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.462576 (+ 237us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":163,"threads_started":1}
W20260925 14:52:36.463218 17150 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35681 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35681: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
I20260925 14:52:36.463554 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.463451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:52761)
0925 14:52:36.463502 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.463535 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:52:36.463940 17152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.463850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:47929)
0925 14:52:36.463894 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.463917 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:52:36.464090 17150 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:52:36.545614 17153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:36.545845 17153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:36.545871 17153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:36.547224 17153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:36.547271 17153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:36.547299 17153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:36.548892 17153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44421
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=40065
--tserver_master_addrs=127.0.98.62:35681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:36.549149 17153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:36.549378 17153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:36.554500 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.554502 17159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:36.554556 17158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:36.555073 17153 server_base.cc:1034] running on GCE node
I20260925 14:52:36.555295 17153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:36.555555 17153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:36.556754 17153 hybrid_clock.cc:648] HybridClock initialized: now 1790347956556749 us; error 40 us; skew 500 ppm
Sep 25 14:52:36 dist-test-slave-z2s5 krb5kdc[15715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:36.559908 17153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:36.560467 17153 webserver.cc:460] Webserver started at http://127.0.98.3:40065/ using document root <none> and password file <none>
I20260925 14:52:36.560685 17153 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:36.560737 17153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:36.562489 17153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:36.563210 17168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.563391 17153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:36.563488 17153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "7a0fa856dc5347b9b63a2c3de72dfada"
format_stamp: "Formatted at 2026-09-25 14:51:55 on dist-test-slave-z2s5"
server_key: "a6855c98fa76585470b989555d6eec2a"
server_key_iv: "e13f1ff88a26f64af8dfe4848f79dc8e"
server_key_version: "encryptionkey@0"
I20260925 14:52:36.563832 17153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:36.575699 17153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:36.578176 17153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:36.578298 17153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:36.578555 17153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:36.578892 17153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:36.578928 17153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.578950 17153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:36.578963 17153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:36.585006 17153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44421
I20260925 14:52:36.585088 17281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44421 every 8 connection(s)
I20260925 14:52:36.585446 17153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:36.586589 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.586121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:55785)
0925 14:52:36.586283 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.586354 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":66,"threads_started":1}
W20260925 14:52:36.587028 17282 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35681 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35681: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
I20260925 14:52:36.587333 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.587239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:60093)
0925 14:52:36.587288 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.587315 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:52:36.587697 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:36.587542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:60909)
0925 14:52:36.587619 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:36.587657 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 14:52:36.587831 17282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:52:36.587867 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17153
I20260925 14:52:37.407362 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.407061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:60189)
0925 14:52:37.407231 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.407284 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
I20260925 14:52:37.464754 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.464538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:42965)
0925 14:52:37.464654 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.464701 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260925 14:52:37.588435 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.588214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:36551)
0925 14:52:37.588348 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.588388 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
I20260925 14:52:37.660914 17289 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:37.661113 17289 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:37.661564 17289 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
Sep 25 14:52:37 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 25 14:52:37 dist-test-slave-z2s5 krb5kdc[15715](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347956, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260925 14:52:37.672580 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.661771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44421 (local address 127.0.98.1:42299)
0925 14:52:37.661833 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.661845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.661901 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.662979 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.662982 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.663365 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:37.663550 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.663555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.664649 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.664650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.665440 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.665441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.666077 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.666078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.666171 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.666817 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:37.666826 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:37.668766 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:37.670880 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.670888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.670890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.671226 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.671491 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.671492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.671607 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.672032 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:37.672038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:37.672181 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.672306 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.672475 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":9}
I20260925 14:52:37.672590 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.661887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42299 (local address 127.0.98.3:44421)
0925 14:52:37.662006 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.662019 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.662052 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:52:37.662716 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.662720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.662783 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.662937 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:37.663685 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.664542 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.665538 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.665989 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.666095 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.668886 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:37.668891 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:37.668891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:37.668901 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:37.670788 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671322 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.671325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.671326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.671412 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671711 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.671712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.671713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.671951 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:37.672041 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.672358 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.672477 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":54,"threads_started":1}
I20260925 14:52:37.672816 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.661937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43751 (local address 127.0.98.2:45995)
0925 14:52:37.662113 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:37.662121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:37.662147 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:52:37.662718 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:37.662720 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:37.662783 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:37.662937 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:37.663685 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.664543 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.665538 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.665934 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.666020 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.668935 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:37.668938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:37.668938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:37.668950 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:37.671299 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671651 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.671654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.671654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.671758 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671979 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:37.671980 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:37.671980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:37.672244 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:37.672340 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:37.672542 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 14:52:37.672679 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260925 14:52:37.672715 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:37.661830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45995 (local address 127.0.98.1:43751)
0925 14:52:37.661994 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:37.662003 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:37.662040 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:37.663067 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:37.663068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:37.663350 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:37.663550 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.663555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.664657 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.664658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.665438 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:37.665440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:37.666015 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:37.666016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:37.666171 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:37.666859 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:37.666881 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:37.668475 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:37.671374 (+ 2899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.671375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.671548 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.671839 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:37.671840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:37.671840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:37.671907 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:37.672337 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:37.672338 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:37.672387 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 14:52:37.672480 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 14:52:37.672591 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":63,"thread_start_us":75,"threads_started":1}
W20260925 14:52:37.674096 16903 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
W20260925 14:52:37.674322 16902 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:37.674360 16902 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba6367128f94bbbaebdc4795dbb59c2; no voters: 7a0fa856dc5347b9b63a2c3de72dfada, b7ed1117709b4ebe8a9b9714c46ce66d
I20260925 14:52:37.674507 17289 raft_consensus.cc:2726] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:52:38.408174 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.407931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:46111)
0925 14:52:38.408086 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:38.408129 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260925 14:52:38.465507 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.465256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:44393)
0925 14:52:38.465425 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:38.465469 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260925 14:52:38.589095 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:38.588909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:38387)
0925 14:52:38.589023 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:38.589060 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260925 14:52:39.409037 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.408747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:47031)
0925 14:52:39.408900 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:39.408979 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260925 14:52:39.466353 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.466105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:48703)
0925 14:52:39.466258 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:39.466304 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260925 14:52:39.589924 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:39.589676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:41245)
0925 14:52:39.589834 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:39.589878 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260925 14:52:39.644886 17299 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:39.645001 17299 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:39.645206 17299 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
W20260925 14:52:39.645630 16903 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
W20260925 14:52:39.645704 16902 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:39.645773 16903 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba6367128f94bbbaebdc4795dbb59c2; no voters: 7a0fa856dc5347b9b63a2c3de72dfada, b7ed1117709b4ebe8a9b9714c46ce66d
I20260925 14:52:39.645840 17299 raft_consensus.cc:2726] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:52:40.409821 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.409606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:50847)
0925 14:52:40.409740 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.409775 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260925 14:52:40.467188 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.466971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:56597)
0925 14:52:40.467100 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.467144 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260925 14:52:40.590636 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:40.590382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:39831)
0925 14:52:40.590556 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:40.590593 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260925 14:52:41.410770 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:41.410457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:48293)
0925 14:52:41.410664 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:41.410718 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
I20260925 14:52:41.467916 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:41.467665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:40263)
0925 14:52:41.467835 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:41.467873 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":60,"threads_started":1}
I20260925 14:52:41.591337 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:41.591119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:36699)
0925 14:52:41.591252 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:41.591294 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260925 14:52:42.411648 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:42.411393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:36657)
0925 14:52:42.411556 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:42.411600 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260925 14:52:42.468760 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:42.468534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:40923)
0925 14:52:42.468672 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:42.468715 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":62,"threads_started":1}
I20260925 14:52:42.563741 17308 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:42.563843 17308 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:42.564018 17308 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
W20260925 14:52:42.564445 16903 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
W20260925 14:52:42.564531 16902 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:42.564603 16902 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba6367128f94bbbaebdc4795dbb59c2; no voters: 7a0fa856dc5347b9b63a2c3de72dfada, b7ed1117709b4ebe8a9b9714c46ce66d
I20260925 14:52:42.564702 17308 raft_consensus.cc:2726] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:52:42.592061 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:42.591830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:56191)
0925 14:52:42.591978 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:42.592020 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":62,"threads_started":1}
I20260925 14:52:43.412432 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:43.412177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:51687)
0925 14:52:43.412352 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:43.412386 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":72,"threads_started":1}
I20260925 14:52:43.469583 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:43.469368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:37321)
0925 14:52:43.469497 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:43.469539 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
I20260925 14:52:43.592837 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:43.592612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:37651)
0925 14:52:43.592755 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:43.592792 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260925 14:52:44.413200 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:44.412989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:46917)
0925 14:52:44.413114 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:44.413154 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 14:52:44.470561 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:44.470155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:59369)
0925 14:52:44.470300 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:44.470337 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260925 14:52:44.593644 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:44.593420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:44591)
0925 14:52:44.593556 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:44.593601 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260925 14:52:45.414114 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:45.413824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:60829)
0925 14:52:45.414016 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:45.414069 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":77,"threads_started":1}
I20260925 14:52:45.471351 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:45.471130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:42503)
0925 14:52:45.471260 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:45.471307 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260925 14:52:45.594477 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:45.594214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:54611)
0925 14:52:45.594367 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:45.594433 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260925 14:52:46.414914 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:46.414690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:38627)
0925 14:52:46.414827 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:46.414870 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260925 14:52:46.472083 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:46.471841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:40027)
0925 14:52:46.471991 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:46.472038 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260925 14:52:46.595315 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:46.595030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:33447)
0925 14:52:46.595194 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:46.595248 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260925 14:52:47.415700 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.415476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:33343)
0925 14:52:47.415607 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:47.415652 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260925 14:52:47.472808 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.472590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:42939)
0925 14:52:47.472723 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:47.472763 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260925 14:52:47.566733 17324 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:47.566828 17324 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:47.566979 17324 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
W20260925 14:52:47.567378 16903 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
W20260925 14:52:47.567560 16902 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:47.567632 16902 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba6367128f94bbbaebdc4795dbb59c2; no voters: 7a0fa856dc5347b9b63a2c3de72dfada, b7ed1117709b4ebe8a9b9714c46ce66d
I20260925 14:52:47.567760 17324 raft_consensus.cc:2726] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:52:47.596050 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:47.595785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:50529)
0925 14:52:47.595956 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:47.596002 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260925 14:52:48.416455 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.416233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:42171)
0925 14:52:48.416376 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:48.416408 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260925 14:52:48.473647 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.473396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:37849)
0925 14:52:48.473552 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:48.473601 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260925 14:52:48.596880 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:48.596665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:36145)
0925 14:52:48.596796 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:48.596837 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
I20260925 14:52:49.417234 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:49.416997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:50991)
0925 14:52:49.417143 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:49.417189 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260925 14:52:49.474484 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:49.474195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:37825)
0925 14:52:49.474322 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:49.474368 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 14:52:49.597716 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:49.597473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:46665)
0925 14:52:49.597623 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:49.597670 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260925 14:52:50.417968 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:50.417752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:50163)
0925 14:52:50.417880 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:50.417920 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260925 14:52:50.475289 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:50.475055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:38759)
0925 14:52:50.475194 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:50.475242 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":67,"threads_started":1}
I20260925 14:52:50.598552 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:50.598240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:46399)
0925 14:52:50.598362 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:50.598452 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260925 14:52:50.969036 17335 raft_consensus.cc:482] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:50.969129 17335 raft_consensus.cc:504] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a0fa856dc5347b9b63a2c3de72dfada" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44421 } } peers { permanent_uuid: "b7ed1117709b4ebe8a9b9714c46ce66d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45995 } } peers { permanent_uuid: "dba6367128f94bbbaebdc4795dbb59c2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40659 } }
I20260925 14:52:50.969336 17335 leader_election.cc:288] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421), b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995)
W20260925 14:52:50.969833 16902 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7a0fa856dc5347b9b63a2c3de72dfada (127.0.98.3:44421): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
W20260925 14:52:50.969907 16903 leader_election.cc:341] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7ed1117709b4ebe8a9b9714c46ce66d (127.0.98.2:45995): Not found: Tablet not found: ead08fac40a74c67999d20e4742c42f9
I20260925 14:52:50.969959 16903 leader_election.cc:302] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dba6367128f94bbbaebdc4795dbb59c2; no voters: 7a0fa856dc5347b9b63a2c3de72dfada, b7ed1117709b4ebe8a9b9714c46ce66d
I20260925 14:52:50.970044 17335 raft_consensus.cc:2726] T ead08fac40a74c67999d20e4742c42f9 P dba6367128f94bbbaebdc4795dbb59c2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 14:52:51.418829 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.418588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.1:44993)
0925 14:52:51.418748 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:51.418786 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":72,"threads_started":1}
I20260925 14:52:51.476162 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.475928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.2:39517)
0925 14:52:51.476075 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:51.476117 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260925 14:52:51.596091 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 16852
I20260925 14:52:51.599345 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:51.599141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35681 (local address 127.0.98.3:57665)
0925 14:52:51.599265 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:51.599305 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
I20260925 14:52:51.600775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17020
I20260925 14:52:51.604998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17153
2026-09-25T14:52:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58305 ms)
[----------] 30 tests from SecurityITest (232565 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:51 dist-test-slave-z2s5 krb5kdc[17341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:51 dist-test-slave-z2s5 krb5kdc[17341](info): set up 2 sockets
Sep 25 14:52:51 dist-test-slave-z2s5 krb5kdc[17341](info): commencing operation
krb5kdc: starting...
W20260925 14:52:52.646127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:52:52 dist-test-slave-z2s5 krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:52:52.770116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41131
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:52.852705 17354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:52.852972 17354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:52.853010 17354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:52.854215 17354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:52.854292 17354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:52.854317 17354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:52.854339 17354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:52.854368 17354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:52.855967 17354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:52.856261 17354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:52.856490 17354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:52.861366 17362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.861414 17360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:52.861414 17359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:52.861783 17354 server_base.cc:1034] running on GCE node
I20260925 14:52:52.862058 17354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:52.862396 17354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:52.863581 17354 hybrid_clock.cc:648] HybridClock initialized: now 1790347972863562 us; error 48 us; skew 500 ppm
Sep 25 14:52:52 dist-test-slave-z2s5 krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:52.866135 17354 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:52.866609 17354 webserver.cc:460] Webserver started at http://127.0.98.62:35527/ using document root <none> and password file <none>
I20260925 14:52:52.866799 17354 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:52.866840 17354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:52.866930 17354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:52.867944 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "ed5e696afd9b4577b11f40c8b4043518"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-z2s5"
server_key: "4ec268d91bac60719e289943bb630ed6"
server_key_iv: "4797266187d95c95c0d1caf59156dc21"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.868268 17354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed5e696afd9b4577b11f40c8b4043518"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-z2s5"
server_key: "4ec268d91bac60719e289943bb630ed6"
server_key_iv: "4797266187d95c95c0d1caf59156dc21"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.869572 17354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:52:52.870325 17369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:52.870616 17354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:52.870692 17354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "ed5e696afd9b4577b11f40c8b4043518"
format_stamp: "Formatted at 2026-09-25 14:52:52 on dist-test-slave-z2s5"
server_key: "4ec268d91bac60719e289943bb630ed6"
server_key_iv: "4797266187d95c95c0d1caf59156dc21"
server_key_version: "encryptionkey@0"
I20260925 14:52:52.870748 17354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:52.896649 17354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:52.899084 17354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:52.899209 17354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:52.903190 17354 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41131
I20260925 14:52:52.903225 17421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41131 every 8 connection(s)
I20260925 14:52:52.903769 17354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:52.904453 17422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:52.905367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17354
I20260925 14:52:52.905498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:52.905690 392 external_mini_cluster.cc:1428] Setting key 64e842f331864a5bb402b369914924fc
I20260925 14:52:52.907886 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518: Bootstrap starting.
I20260925 14:52:52.908654 17422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:52.908977 17422 log.cc:824] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:52.909955 17422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518: No bootstrap required, opened a new log
I20260925 14:52:52.912386 17422 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5e696afd9b4577b11f40c8b4043518" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } }
I20260925 14:52:52.912580 17422 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:52.912632 17422 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5e696afd9b4577b11f40c8b4043518, State: Initialized, Role: FOLLOWER
I20260925 14:52:52.912804 17422 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5e696afd9b4577b11f40c8b4043518" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } }
I20260925 14:52:52.912919 17422 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:52.912959 17422 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:52.913017 17422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 14:52:52 dist-test-slave-z2s5 krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347972, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:52.913708 17422 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5e696afd9b4577b11f40c8b4043518" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } }
I20260925 14:52:52.913841 17422 leader_election.cc:302] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5e696afd9b4577b11f40c8b4043518; no voters:
I20260925 14:52:52.914050 17422 leader_election.cc:288] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:52.914084 17427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:52.914275 17427 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [term 1 LEADER]: Becoming Leader. State: Replica: ed5e696afd9b4577b11f40c8b4043518, State: Running, Role: LEADER
I20260925 14:52:52.914433 17427 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5e696afd9b4577b11f40c8b4043518" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } }
I20260925 14:52:52.914448 17422 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:52.915761 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5e696afd9b4577b11f40c8b4043518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5e696afd9b4577b11f40c8b4043518" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } }
I20260925 14:52:52.915920 17429 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5e696afd9b4577b11f40c8b4043518. Latest consensus state: current_term: 1 leader_uuid: "ed5e696afd9b4577b11f40c8b4043518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5e696afd9b4577b11f40c8b4043518" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41131 } } }
I20260925 14:52:52.916033 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:52.916042 17429 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:52.916326 17443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:52.916855 17443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:52.917908 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:52.906246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58050 (local address 127.0.98.62:41131)
0925 14:52:52.906466 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:52.906475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:52.906516 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:52:52.907066 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:52.907076 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:52.907144 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:52.907302 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:52.907906 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.909332 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.910301 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:52.910979 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:52.911114 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:52.913509 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:52.913514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:52.913514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:52.913528 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:52.915930 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.916441 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.916444 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.916444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.916566 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:52.916826 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:52.916827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:52.916827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:52.917148 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:52.917312 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:52.917442 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:52:52.917698 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260925 14:52:52.918531 17443 catalog_manager.cc:1347] Generated new cluster ID: 9dcac351ef66439392dd2fd9834297cd
I20260925 14:52:52.918638 17443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:52.928857 17443 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:52.929565 17443 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:52.941746 17443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518: Generated new TSK 0
I20260925 14:52:52.941998 17443 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:52:52.976953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41131
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:53.075505 17449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.075726 17449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.075752 17449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.076903 17449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.076941 17449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.076967 17449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:53.078377 17449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:53.078583 17449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.078759 17449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:52:53.083822 17449 server_base.cc:1034] running on GCE node
W20260925 14:52:53.083734 17455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.083738 17454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.083734 17457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.084252 17449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.084493 17449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.085673 17449 hybrid_clock.cc:648] HybridClock initialized: now 1790347973085668 us; error 36 us; skew 500 ppm
Sep 25 14:52:53 dist-test-slave-z2s5 krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:53.088145 17449 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:53.088810 17449 webserver.cc:460] Webserver started at http://127.0.98.1:33853/ using document root <none> and password file <none>
I20260925 14:52:53.089097 17449 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.089174 17449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.089327 17449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:53.090508 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "3af7be567a7a4fb5848ef328c579b307"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "19cb1559184e588cb5c2f551f9f2d1fa"
server_key_iv: "f1a4bb949c68b5afc681ae3b2faddb91"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.090845 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3af7be567a7a4fb5848ef328c579b307"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "19cb1559184e588cb5c2f551f9f2d1fa"
server_key_iv: "f1a4bb949c68b5afc681ae3b2faddb91"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.092116 17449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.092780 17464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.092931 17449 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:53.093031 17449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "3af7be567a7a4fb5848ef328c579b307"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "19cb1559184e588cb5c2f551f9f2d1fa"
server_key_iv: "f1a4bb949c68b5afc681ae3b2faddb91"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.093101 17449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.102701 17449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.104494 17449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.104662 17449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.104882 17449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:53.105189 17449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:53.105243 17449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.105285 17449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:53.105312 17449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.111189 17449 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42677
I20260925 14:52:53.111235 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42677 every 8 connection(s)
I20260925 14:52:53.111554 17449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:53.112474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17449
I20260925 14:52:53.112591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:53.112802 392 external_mini_cluster.cc:1428] Setting key 33e13f73326472a69fe8df7bd3d8fbd0
Sep 25 14:52:53 dist-test-slave-z2s5 krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:52:53.123045 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.112226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51419 (local address 127.0.98.62:41131)
0925 14:52:53.112293 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.112296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.112620 (+ 324us) server_negotiation.cc:407] Connection header received
0925 14:52:53.113874 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.113874 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.113884 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.113936 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.114863 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.115398 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.116654 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.116865 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.116973 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.119818 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.119819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.119820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.119826 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.121325 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.121840 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.121841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.121841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.121903 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.122209 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.122210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.122210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.122316 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.122368 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.122892 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.122978 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:52:53.123242 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.112259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41131 (local address 127.0.98.1:51419)
0925 14:52:53.112468 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.112527 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.113750 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.114013 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.114018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.114468 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.114765 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.114771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.115495 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.115496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.116555 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.116558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.116988 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.116989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.117130 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.117806 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.117817 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.119705 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.121417 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.121424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.121426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.121729 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.121998 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.121999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.122000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.122147 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.122518 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.122524 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.122668 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.122846 (+ 178us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.122989 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260925 14:52:53.123842 17578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41131
I20260925 14:52:53.123939 17578 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.124099 17578 heartbeater.cc:511] Master 127.0.98.62:41131 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260925 14:52:53.124691 17386 ts_manager.cc:194] Registered new tserver with Master: 3af7be567a7a4fb5848ef328c579b307 (127.0.98.1:42677)
I20260925 14:52:53.125296 17386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:52:53.140759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41131
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:53.223304 17583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.223539 17583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.223567 17583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.224843 17583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.224902 17583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.224972 17583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:53.226547 17583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:53.226745 17583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.226930 17583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.232179 17589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.232182 17591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.232201 17583 server_base.cc:1034] running on GCE node
W20260925 14:52:53.232254 17588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.232582 17583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.232836 17583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.234010 17583 hybrid_clock.cc:648] HybridClock initialized: now 1790347973233996 us; error 45 us; skew 500 ppm
Sep 25 14:52:53 dist-test-slave-z2s5 krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:53.236582 17583 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:53.237073 17583 webserver.cc:460] Webserver started at http://127.0.98.2:46123/ using document root <none> and password file <none>
I20260925 14:52:53.237298 17583 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.237365 17583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.237516 17583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:53.238653 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "12899f1f4cbd40f7a44667c5c1a45e12"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "82bf704ebf78fd4702f35405d22da08b"
server_key_iv: "e5d247923b8a4fbf5c456be49f9ceb75"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.239023 17583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12899f1f4cbd40f7a44667c5c1a45e12"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "82bf704ebf78fd4702f35405d22da08b"
server_key_iv: "e5d247923b8a4fbf5c456be49f9ceb75"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.240393 17583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:53.241194 17598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.241411 17583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.241565 17583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "12899f1f4cbd40f7a44667c5c1a45e12"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "82bf704ebf78fd4702f35405d22da08b"
server_key_iv: "e5d247923b8a4fbf5c456be49f9ceb75"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.241647 17583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.267756 17583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.270161 17583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.270354 17583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.270682 17583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:53.271024 17583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:53.271082 17583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.271126 17583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:53.271155 17583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.277426 17583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46465
I20260925 14:52:53.277465 17711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46465 every 8 connection(s)
I20260925 14:52:53.277820 17583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:52:53 dist-test-slave-z2s5 krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:53.285821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17583
I20260925 14:52:53.285941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:53.286172 392 external_mini_cluster.cc:1428] Setting key a8955a649552d76d28d97e2ff8078aa1
I20260925 14:52:53.287912 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.278616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41131 (local address 127.0.98.2:54705)
0925 14:52:53.278793 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.278831 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.279777 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.280163 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.280166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.280423 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.280615 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.280618 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.281295 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.281296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.282113 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.282115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.282514 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.282516 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.282630 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.283104 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.283113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.284538 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.286281 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.286288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.286290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.286646 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.286854 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.286855 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.286855 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.286998 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.287382 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.287387 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.287513 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.287605 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.287748 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260925 14:52:53.287918 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.278632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54705 (local address 127.0.98.62:41131)
0925 14:52:53.278701 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.278703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.278924 (+ 221us) server_negotiation.cc:407] Connection header received
0925 14:52:53.279935 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.279936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.279949 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.279982 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.280700 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.281210 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.282194 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.282378 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.282489 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.284647 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.284649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.284649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.284655 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.286189 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.286728 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.286729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.286729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.286782 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.287070 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.287072 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.287072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.287272 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.287367 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.287653 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.287764 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:52:53.288436 17712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41131
I20260925 14:52:53.288569 17712 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.288796 17712 heartbeater.cc:511] Master 127.0.98.62:41131 requested a full tablet report, sending...
I20260925 14:52:53.289191 17386 ts_manager.cc:194] Registered new tserver with Master: 12899f1f4cbd40f7a44667c5c1a45e12 (127.0.98.2:46465)
I20260925 14:52:53.289579 17386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54705
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:52:53.314127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41131
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:53.395507 17717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.395754 17717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.395792 17717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.397001 17717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.397068 17717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.397112 17717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:53.398597 17717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:53.398874 17717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.399113 17717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.404042 17722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.404016 17725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.404142 17717 server_base.cc:1034] running on GCE node
W20260925 14:52:53.404021 17723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.404460 17717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.404717 17717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.405880 17717 hybrid_clock.cc:648] HybridClock initialized: now 1790347973405870 us; error 45 us; skew 500 ppm
Sep 25 14:52:53 dist-test-slave-z2s5 krb5kdc[17341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:53.408493 17717 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:53.408999 17717 webserver.cc:460] Webserver started at http://127.0.98.3:36063/ using document root <none> and password file <none>
I20260925 14:52:53.409246 17717 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.409296 17717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.409430 17717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:53.410615 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "9b50f8f4496a4e58916903e953fcd471"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "f9d981e6baf927911193924f68ba8dce"
server_key_iv: "4a0dac92014ed40c2100b0ac15d5408f"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.410974 17717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9b50f8f4496a4e58916903e953fcd471"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "f9d981e6baf927911193924f68ba8dce"
server_key_iv: "4a0dac92014ed40c2100b0ac15d5408f"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.412299 17717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.413067 17732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.413280 17717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.413380 17717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "9b50f8f4496a4e58916903e953fcd471"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "f9d981e6baf927911193924f68ba8dce"
server_key_iv: "4a0dac92014ed40c2100b0ac15d5408f"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.413460 17717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.430128 17717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.432570 17717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.432750 17717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.432993 17717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:53.433308 17717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:53.433363 17717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.433408 17717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:53.433435 17717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.439407 17717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39299
I20260925 14:52:53.439450 17845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39299 every 8 connection(s)
I20260925 14:52:53.439805 17717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:52:53 dist-test-slave-z2s5 krb5kdc[17341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:53.449327 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17717
I20260925 14:52:53.449417 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:53.449640 392 external_mini_cluster.cc:1428] Setting key d3f3abcc90d30dbb3bb9b8654290a7e4
I20260925 14:52:53.449944 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.440595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39301 (local address 127.0.98.62:41131)
0925 14:52:53.440695 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.440697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.440884 (+ 187us) server_negotiation.cc:407] Connection header received
0925 14:52:53.441881 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.441883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.441896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.441952 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.442696 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.443210 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.444249 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.444544 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.444621 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.446991 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.446993 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.446993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.447001 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.448472 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.448893 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.448894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.448895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.448961 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.449223 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.449224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.449225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.449352 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.449436 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.449769 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.449850 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:52:53.450165 17848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.440581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41131 (local address 127.0.98.3:39301)
0925 14:52:53.440752 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.440796 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.441751 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.442032 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.442036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.442324 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.442596 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.442601 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.443313 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.443314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.444146 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.444148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.444630 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.444631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.444778 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.445361 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:53.445370 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.446880 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.448575 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.448582 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.448585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.448814 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.449029 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:53.449030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:53.449031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:53.449147 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:53.449469 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.449475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.449621 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.449745 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.449910 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260925 14:52:53.450703 17846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41131
I20260925 14:52:53.450802 17846 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.450947 17846 heartbeater.cc:511] Master 127.0.98.62:41131 requested a full tablet report, sending...
I20260925 14:52:53.451318 17386 ts_manager.cc:194] Registered new tserver with Master: 9b50f8f4496a4e58916903e953fcd471 (127.0.98.3:39299)
I20260925 14:52:53.451768 17386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39301
I20260925 14:52:53.461885 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:53.469213 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.463412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58054 (local address 127.0.98.62:41131)
0925 14:52:53.463536 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.463538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.463549 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:52:53.463604 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.463605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.463615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.463681 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.464189 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.464694 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.465393 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.465595 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.465656 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.466939 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.466940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.466940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:53.466946 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.468260 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.468564 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.468564 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.468565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.468617 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:53.468793 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:53.468794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:53.468794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:53.468929 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.469017 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.469088 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.469149 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 14:52:53.470969 17386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:53.472499 17386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:53.479532 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.475075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39518 (local address 127.0.98.1:42677)
0925 14:52:53.475833 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.475838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.475847 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:52:53.475884 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.475886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.475901 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.475980 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:53.476151 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.476837 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.478458 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.479141 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.479215 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.479264 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.479396 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.479442 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":77,"threads_started":1}
I20260925 14:52:53.479560 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.474926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42677 (local address 127.0.0.1:39518)
0925 14:52:53.475238 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.475247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.475287 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.475979 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.475980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.475982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:53.476066 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.476069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.476943 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.476944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.478360 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.478363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.479216 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.479217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.479276 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.479288 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.479371 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.479451 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":72,"thread_start_us":84,"threads_started":1}
I20260925 14:52:53.479835 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.475073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39766 (local address 127.0.98.3:39299)
0925 14:52:53.475480 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.475484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.475494 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:53.475585 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.475587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.475605 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.475682 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:53.475899 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.477008 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.478267 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.479064 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.479147 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.479217 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.479710 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.479755 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":74,"threads_started":1}
I20260925 14:52:53.480276 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.474891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39299 (local address 127.0.0.1:39766)
0925 14:52:53.475054 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.475213 (+ 159us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.475287 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.475772 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.475773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.475775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:53.475818 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.475819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.477114 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.477115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.478159 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.478161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.479163 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.479163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.479272 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.479288 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.479371 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.479451 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":34,"thread_start_us":74,"threads_started":1}
I20260925 14:52:53.480651 17512 tablet_service.cc:1467] Processing CreateTablet for tablet 442feeacf1a24999b22d9ab191ac5b8a (DEFAULT_TABLE table=test-table [id=57ab4a44e61b45cbbd33522bd5de7d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:53.481003 17512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442feeacf1a24999b22d9ab191ac5b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:53.480997 17780 tablet_service.cc:1467] Processing CreateTablet for tablet 442feeacf1a24999b22d9ab191ac5b8a (DEFAULT_TABLE table=test-table [id=57ab4a44e61b45cbbd33522bd5de7d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:53.481326 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442feeacf1a24999b22d9ab191ac5b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:53.483893 17863 tablet_bootstrap.cc:492] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307: Bootstrap starting.
I20260925 14:52:53.484041 17864 tablet_bootstrap.cc:492] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Bootstrap starting.
I20260925 14:52:53.484638 17864 tablet_bootstrap.cc:654] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:53.484665 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.475114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46465 (local address 127.0.0.1:52276)
0925 14:52:53.475384 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.475392 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.475423 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.475650 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.475654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.475656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:53.475744 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.475746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.477099 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.477100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.478363 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.478364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.484371 (+ 6007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.484372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.484450 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.484456 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.484506 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.484564 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":61,"thread_start_us":73,"threads_started":1}
I20260925 14:52:53.484658 17863 tablet_bootstrap.cc:654] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:53.485015 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.475273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.2:46465)
0925 14:52:53.475428 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.475433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.475475 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:52:53.475524 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.475526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.475542 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.475632 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:53.475835 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.476981 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.479310 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.484154 (+ 4844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.484249 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.484328 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.484847 (+ 519us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.484916 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
I20260925 14:52:53.485242 17864 log.cc:824] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:53.485268 17863 log.cc:824] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:53.485743 17646 tablet_service.cc:1467] Processing CreateTablet for tablet 442feeacf1a24999b22d9ab191ac5b8a (DEFAULT_TABLE table=test-table [id=57ab4a44e61b45cbbd33522bd5de7d67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:53.486016 17863 tablet_bootstrap.cc:492] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307: No bootstrap required, opened a new log
I20260925 14:52:53.486030 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 442feeacf1a24999b22d9ab191ac5b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:53.486038 17864 tablet_bootstrap.cc:492] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: No bootstrap required, opened a new log
I20260925 14:52:53.486080 17863 ts_tablet_manager.cc:1397] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:53.486086 17864 ts_tablet_manager.cc:1397] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:53.488080 17863 raft_consensus.cc:348] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.488077 17864 raft_consensus.cc:348] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.488265 17863 raft_consensus.cc:374] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:53.488265 17864 raft_consensus.cc:374] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:53.488303 17864 raft_consensus.cc:729] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b50f8f4496a4e58916903e953fcd471, State: Initialized, Role: FOLLOWER
I20260925 14:52:53.488302 17863 raft_consensus.cc:729] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af7be567a7a4fb5848ef328c579b307, State: Initialized, Role: FOLLOWER
I20260925 14:52:53.488422 17864 consensus_queue.cc:260] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.488422 17863 consensus_queue.cc:260] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.488593 17867 tablet_bootstrap.cc:492] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Bootstrap starting.
I20260925 14:52:53.488698 17863 ts_tablet_manager.cc:1428] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:53.488698 17864 ts_tablet_manager.cc:1428] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:53.488833 17846 heartbeater.cc:503] Master 127.0.98.62:41131 was elected leader, sending a full tablet report...
I20260925 14:52:53.488832 17578 heartbeater.cc:503] Master 127.0.98.62:41131 was elected leader, sending a full tablet report...
I20260925 14:52:53.489568 17867 tablet_bootstrap.cc:654] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:53.490088 17867 log.cc:824] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:53.490933 17867 tablet_bootstrap.cc:492] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: No bootstrap required, opened a new log
I20260925 14:52:53.490989 17867 ts_tablet_manager.cc:1397] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:53.492311 17867 raft_consensus.cc:348] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.492450 17867 raft_consensus.cc:374] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:53.492475 17867 raft_consensus.cc:729] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12899f1f4cbd40f7a44667c5c1a45e12, State: Initialized, Role: FOLLOWER
I20260925 14:52:53.492595 17867 consensus_queue.cc:260] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.492892 17867 ts_tablet_manager.cc:1428] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:53.492904 17712 heartbeater.cc:503] Master 127.0.98.62:41131 was elected leader, sending a full tablet report...
I20260925 14:52:53.508968 17869 raft_consensus.cc:482] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:53.509109 17869 raft_consensus.cc:504] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.509584 17869 leader_election.cc:288] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b50f8f4496a4e58916903e953fcd471 (127.0.98.3:39299), 12899f1f4cbd40f7a44667c5c1a45e12 (127.0.98.2:46465)
I20260925 14:52:53.513670 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.509838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41955 (local address 127.0.98.3:39299)
0925 14:52:53.509921 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.509923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.509933 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:53.509961 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.509962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.509978 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.510090 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:53.510301 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.511093 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.512679 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.513253 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.513324 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.513334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.513551 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.513596 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:52:53.513670 17862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.509968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47877 (local address 127.0.98.2:46465)
0925 14:52:53.510046 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.510048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.510143 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:52:53.510184 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.510185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.510194 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.510232 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:53.510415 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.511096 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.512679 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.513253 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.513324 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.513334 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.513549 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.513596 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 14:52:53.513674 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.509731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39299 (local address 127.0.98.1:41955)
0925 14:52:53.509793 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.509806 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.509851 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.510109 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.510110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.510112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:53.510206 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.510208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.511194 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.511195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.512575 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.512579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.513373 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.513375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.513459 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.513462 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.513530 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.513567 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":62}
I20260925 14:52:53.513679 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.509871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46465 (local address 127.0.98.1:47877)
0925 14:52:53.510048 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.510078 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.510133 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.510254 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.510255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.510257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:53.510331 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.510332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.511185 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.511186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.512575 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.512579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.513374 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.513375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.513459 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.513463 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.513522 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.513566 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":112,"thread_start_us":111,"threads_started":1}
I20260925 14:52:53.514171 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442feeacf1a24999b22d9ab191ac5b8a" candidate_uuid: "3af7be567a7a4fb5848ef328c579b307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b50f8f4496a4e58916903e953fcd471" is_pre_election: true
I20260925 14:52:53.514274 17666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442feeacf1a24999b22d9ab191ac5b8a" candidate_uuid: "3af7be567a7a4fb5848ef328c579b307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" is_pre_election: true
I20260925 14:52:53.514344 17800 raft_consensus.cc:2454] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af7be567a7a4fb5848ef328c579b307 in term 0.
I20260925 14:52:53.514422 17666 raft_consensus.cc:2454] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af7be567a7a4fb5848ef328c579b307 in term 0.
I20260925 14:52:53.514605 17465 leader_election.cc:302] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af7be567a7a4fb5848ef328c579b307, 9b50f8f4496a4e58916903e953fcd471; no voters:
I20260925 14:52:53.514773 17869 raft_consensus.cc:2781] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:53.514854 17869 raft_consensus.cc:482] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:53.514874 17869 raft_consensus.cc:3037] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:53.515839 17869 raft_consensus.cc:504] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.515995 17869 leader_election.cc:288] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [CANDIDATE]: Term 1 election: Requested vote from peers 9b50f8f4496a4e58916903e953fcd471 (127.0.98.3:39299), 12899f1f4cbd40f7a44667c5c1a45e12 (127.0.98.2:46465)
I20260925 14:52:53.516348 17666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442feeacf1a24999b22d9ab191ac5b8a" candidate_uuid: "3af7be567a7a4fb5848ef328c579b307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12899f1f4cbd40f7a44667c5c1a45e12"
I20260925 14:52:53.516438 17666 raft_consensus.cc:3037] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:53.517171 17666 raft_consensus.cc:2454] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af7be567a7a4fb5848ef328c579b307 in term 1.
I20260925 14:52:53.517298 17800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "442feeacf1a24999b22d9ab191ac5b8a" candidate_uuid: "3af7be567a7a4fb5848ef328c579b307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b50f8f4496a4e58916903e953fcd471"
I20260925 14:52:53.517328 17465 leader_election.cc:302] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12899f1f4cbd40f7a44667c5c1a45e12, 3af7be567a7a4fb5848ef328c579b307; no voters:
I20260925 14:52:53.517400 17800 raft_consensus.cc:3037] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:53.517421 17869 raft_consensus.cc:2781] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:53.517597 17869 raft_consensus.cc:686] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [term 1 LEADER]: Becoming Leader. State: Replica: 3af7be567a7a4fb5848ef328c579b307, State: Running, Role: LEADER
I20260925 14:52:53.517704 17869 consensus_queue.cc:237] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } }
I20260925 14:52:53.518337 17800 raft_consensus.cc:2454] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af7be567a7a4fb5848ef328c579b307 in term 1.
I20260925 14:52:53.518785 17386 catalog_manager.cc:5495] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3af7be567a7a4fb5848ef328c579b307 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3af7be567a7a4fb5848ef328c579b307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af7be567a7a4fb5848ef328c579b307" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:53.527283 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.524869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.0.98.1:42677)
0925 14:52:53.524920 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.524922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.524930 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:53.524966 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.524966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.524987 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.525076 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:53.525206 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.525668 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.526464 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.526677 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.526756 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.526929 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:53.527035 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:53.527131 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.527150 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.527171 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 14:52:53.528415 17713 tablet.cc:2369] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:53.529193 17800 raft_consensus.cc:1261] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 1 FOLLOWER]: Refusing update from remote peer 3af7be567a7a4fb5848ef328c579b307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:53.529217 17666 raft_consensus.cc:1261] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 1 FOLLOWER]: Refusing update from remote peer 3af7be567a7a4fb5848ef328c579b307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:53.529558 17873 consensus_queue.cc:1035] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b50f8f4496a4e58916903e953fcd471" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:53.529722 17869 consensus_queue.cc:1035] T 442feeacf1a24999b22d9ab191ac5b8a P 3af7be567a7a4fb5848ef328c579b307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12899f1f4cbd40f7a44667c5c1a45e12" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:53.532116 17876 mvcc.cc:205] Tried to move back new op lower bound from 7333265299572588544 to 7333265299529166848. Current Snapshot: MvccSnapshot[applied={T|T < 7333265299572588544}]
I20260925 14:52:53.533074 17878 mvcc.cc:205] Tried to move back new op lower bound from 7333265299572588544 to 7333265299529166848. Current Snapshot: MvccSnapshot[applied={T|T < 7333265299572588544}]
I20260925 14:52:53.534323 17384 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:53.534448 17384 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:53.535187 17384 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed5e696afd9b4577b11f40c8b4043518: Sending DeleteTablet for 3 replicas of tablet 442feeacf1a24999b22d9ab191ac5b8a
I20260925 14:52:53.535506 17780 tablet_service.cc:1514] Processing DeleteTablet for tablet 442feeacf1a24999b22d9ab191ac5b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39766
I20260925 14:52:53.535542 17646 tablet_service.cc:1514] Processing DeleteTablet for tablet 442feeacf1a24999b22d9ab191ac5b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52276
I20260925 14:52:53.535706 17886 tablet_replica.cc:330] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: stopping tablet replica
I20260925 14:52:53.535734 17512 tablet_service.cc:1514] Processing DeleteTablet for tablet 442feeacf1a24999b22d9ab191ac5b8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39518
I20260925 14:52:53.535800 17886 raft_consensus.cc:2229] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:53.535804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17449
I20260925 14:52:53.535892 17886 raft_consensus.cc:2258] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:53.538682 17887 tablet_replica.cc:330] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: stopping tablet replica
I20260925 14:52:53.538767 17886 ts_tablet_manager.cc:1905] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:53.538794 17887 raft_consensus.cc:2229] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:53.538908 17887 raft_consensus.cc:2258] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:53.539336 17887 ts_tablet_manager.cc:1905] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:53.540850 17886 ts_tablet_manager.cc:1918] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:53.540889 17887 ts_tablet_manager.cc:1918] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:53.540938 17886 log.cc:1196] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/442feeacf1a24999b22d9ab191ac5b8a
I20260925 14:52:53.540942 17887 log.cc:1196] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/442feeacf1a24999b22d9ab191ac5b8a
I20260925 14:52:53.541229 17887 ts_tablet_manager.cc:1939] T 442feeacf1a24999b22d9ab191ac5b8a P 12899f1f4cbd40f7a44667c5c1a45e12: Deleting consensus metadata
I20260925 14:52:53.541240 17886 ts_tablet_manager.cc:1939] T 442feeacf1a24999b22d9ab191ac5b8a P 9b50f8f4496a4e58916903e953fcd471: Deleting consensus metadata
I20260925 14:52:53.541770 17370 catalog_manager.cc:4841] TS 9b50f8f4496a4e58916903e953fcd471 (127.0.98.3:39299): tablet 442feeacf1a24999b22d9ab191ac5b8a (table test-table [id=57ab4a44e61b45cbbd33522bd5de7d67]) successfully deleted
I20260925 14:52:53.541883 17370 catalog_manager.cc:4841] TS 12899f1f4cbd40f7a44667c5c1a45e12 (127.0.98.2:46465): tablet 442feeacf1a24999b22d9ab191ac5b8a (table test-table [id=57ab4a44e61b45cbbd33522bd5de7d67]) successfully deleted
W20260925 14:52:53.541806 17373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42677 (error 108)
I20260925 14:52:53.542030 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17583
I20260925 14:52:53.542665 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.542322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42677 (local address 127.0.0.1:39546)
0925 14:52:53.542579 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.542624 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234}
W20260925 14:52:53.542805 17373 catalog_manager.cc:4584] TS 3af7be567a7a4fb5848ef328c579b307 (127.0.98.1:42677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 442feeacf1a24999b22d9ab191ac5b8a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42677: connect: Connection refused (error 111)
I20260925 14:52:53.547590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17717
I20260925 14:52:53.554841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17354
2026-09-25T14:52:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1950 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T14:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:53Z Disabled control of system clock
I20260925 14:52:53.583271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38805
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:53.663920 17893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.664156 17893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.664179 17893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.665398 17893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:53.665444 17893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.665462 17893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.665475 17893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:53.665488 17893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:53.666965 17893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38805
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:53.667243 17893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.667553 17893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.672430 17898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.672438 17901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.672437 17899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.672776 17893 server_base.cc:1034] running on GCE node
I20260925 14:52:53.673048 17893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.673347 17893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.674458 17893 hybrid_clock.cc:648] HybridClock initialized: now 1790347973674434 us; error 41 us; skew 500 ppm
I20260925 14:52:53.675663 17893 webserver.cc:460] Webserver started at http://127.0.98.62:39619/ using document root <none> and password file <none>
I20260925 14:52:53.675913 17893 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.675983 17893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.676139 17893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:53.677232 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "9c6d454741914b6fa4c5a121f8431cc5"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "d63c1b13df9e6dd70eb771b824b486c7"
server_key_iv: "295e29221fc8ad2567899b2f0c482721"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.677610 17893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c6d454741914b6fa4c5a121f8431cc5"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "d63c1b13df9e6dd70eb771b824b486c7"
server_key_iv: "295e29221fc8ad2567899b2f0c482721"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.679083 17893 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:52:53.679898 17907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.680135 17893 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.680263 17893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "9c6d454741914b6fa4c5a121f8431cc5"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "d63c1b13df9e6dd70eb771b824b486c7"
server_key_iv: "295e29221fc8ad2567899b2f0c482721"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.680346 17893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.689589 17893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.689914 17893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.690064 17893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.694473 17893 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38805
I20260925 14:52:53.694510 17959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38805 every 8 connection(s)
I20260925 14:52:53.694859 17893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:53.695859 17960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:53.697723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17893
I20260925 14:52:53.697829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:53.698024 392 external_mini_cluster.cc:1428] Setting key fc163139f5b447fd249d5b920e9eaced
I20260925 14:52:53.698164 17960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5: Bootstrap starting.
I20260925 14:52:53.698921 17960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:53.699229 17960 log.cc:824] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:53.699909 17960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5: No bootstrap required, opened a new log
I20260925 14:52:53.701658 17960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6d454741914b6fa4c5a121f8431cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38805 } }
I20260925 14:52:53.701813 17960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:53.701859 17960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6d454741914b6fa4c5a121f8431cc5, State: Initialized, Role: FOLLOWER
I20260925 14:52:53.702030 17960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6d454741914b6fa4c5a121f8431cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38805 } }
I20260925 14:52:53.702127 17960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:53.702167 17960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:53.702215 17960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:53.702816 17960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6d454741914b6fa4c5a121f8431cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38805 } }
I20260925 14:52:53.702934 17960 leader_election.cc:302] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c6d454741914b6fa4c5a121f8431cc5; no voters:
I20260925 14:52:53.703095 17960 leader_election.cc:288] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:53.703190 17965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:53.703365 17960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:53.703370 17965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6d454741914b6fa4c5a121f8431cc5, State: Running, Role: LEADER
I20260925 14:52:53.703698 17965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6d454741914b6fa4c5a121f8431cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38805 } }
I20260925 14:52:53.703871 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.698578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.0.98.62:38805)
0925 14:52:53.698805 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.698812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.698838 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:52:53.699293 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.699301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.699402 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.699553 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.699767 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.700710 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.701498 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.701994 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.702110 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.703027 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.703031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.703031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:53.703248 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.703292 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:53.703306 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.703414 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.703506 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.703587 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260925 14:52:53.704097 17967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c6d454741914b6fa4c5a121f8431cc5. Latest consensus state: current_term: 1 leader_uuid: "9c6d454741914b6fa4c5a121f8431cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6d454741914b6fa4c5a121f8431cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38805 } } }
I20260925 14:52:53.704175 17967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.704475 17966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c6d454741914b6fa4c5a121f8431cc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6d454741914b6fa4c5a121f8431cc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38805 } } }
I20260925 14:52:53.704530 17966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:53.704770 17974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:53.705381 17974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:53.706790 17974 catalog_manager.cc:1347] Generated new cluster ID: 54bcd2be5e404d87a63496113136d293
I20260925 14:52:53.706856 17974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:53.718165 17974 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:53.718801 17974 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:53.723465 17974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5: Generated new TSK 0
I20260925 14:52:53.723708 17974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:53.725898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38805
--builtin_ntp_servers=127.0.98.20:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:53.815683 17984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.815887 17984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.815909 17984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.817060 17984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.817101 17984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.817133 17984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:53.818480 17984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:53.818724 17984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.819031 17984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.823974 17989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.824013 17992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.824261 17990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.824810 17984 server_base.cc:1034] running on GCE node
I20260925 14:52:53.825045 17984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.825297 17984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.826460 17984 hybrid_clock.cc:648] HybridClock initialized: now 1790347973826427 us; error 49 us; skew 500 ppm
I20260925 14:52:53.827750 17984 webserver.cc:460] Webserver started at http://127.0.98.1:41701/ using document root <none> and password file <none>
I20260925 14:52:53.827991 17984 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.828063 17984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.828236 17984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:53.829334 17984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "523bf94fa4b340d5a6f87e1554005e9c"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "24dfa77ec3ba6c4c6ed73913499b5d59"
server_key_iv: "f0be373fcb43240630e747ff20bd6579"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.829702 17984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "523bf94fa4b340d5a6f87e1554005e9c"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "24dfa77ec3ba6c4c6ed73913499b5d59"
server_key_iv: "f0be373fcb43240630e747ff20bd6579"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.831220 17984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:52:53.832033 17998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.832229 17984 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:52:53.832333 17984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "523bf94fa4b340d5a6f87e1554005e9c"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "24dfa77ec3ba6c4c6ed73913499b5d59"
server_key_iv: "f0be373fcb43240630e747ff20bd6579"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.832412 17984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.849644 17984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.849988 17984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.850143 17984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.850440 17984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:53.850796 17984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:53.850853 17984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.850895 17984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:53.850921 17984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.857365 17984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38261
I20260925 14:52:53.857409 18111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38261 every 8 connection(s)
I20260925 14:52:53.857766 17984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:53.861354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 17984
I20260925 14:52:53.861456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:53.861637 392 external_mini_cluster.cc:1428] Setting key 0ef58d54e990466644fd133963b17773
I20260925 14:52:53.862785 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38805
--builtin_ntp_servers=127.0.98.20:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:52:53.863741 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.858522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52187 (local address 127.0.98.62:38805)
0925 14:52:53.858567 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.858569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.858965 (+ 396us) server_negotiation.cc:407] Connection header received
0925 14:52:53.859467 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.859468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.859479 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.859546 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.860193 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.860733 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.862068 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.862370 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.862471 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.863477 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.863479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.863479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:53.863484 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.863499 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:53.863503 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.863562 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.863655 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.863681 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:52:53.864048 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.858598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38805 (local address 127.0.98.1:52187)
0925 14:52:53.858823 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.858860 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.859362 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.859606 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.859609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.859617 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.860120 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.860123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.860819 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.860820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.861972 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.861976 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.862485 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.862486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.862654 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.863310 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:53.863311 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.863352 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:53.863385 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.863570 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.863576 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.863578 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.863649 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.863767 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":112,"threads_started":1}
I20260925 14:52:53.864545 18112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38805
I20260925 14:52:53.864640 18112 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.864863 18112 heartbeater.cc:511] Master 127.0.98.62:38805 requested a full tablet report, sending...
I20260925 14:52:53.865386 17924 ts_manager.cc:194] Registered new tserver with Master: 523bf94fa4b340d5a6f87e1554005e9c (127.0.98.1:38261)
I20260925 14:52:53.866020 17924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52187
W20260925 14:52:53.944406 18115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:53.944661 18115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:53.944698 18115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:53.945924 18115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:53.946000 18115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:53.946053 18115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:53.947628 18115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:53.947871 18115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:53.948237 18115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:53.953261 18121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.953271 18123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:53.953307 18120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:53.953552 18115 server_base.cc:1034] running on GCE node
I20260925 14:52:53.953819 18115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:53.954037 18115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:53.955206 18115 hybrid_clock.cc:648] HybridClock initialized: now 1790347973955196 us; error 28 us; skew 500 ppm
I20260925 14:52:53.956696 18115 webserver.cc:460] Webserver started at http://127.0.98.2:35149/ using document root <none> and password file <none>
I20260925 14:52:53.956943 18115 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:53.956993 18115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:53.957147 18115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:53.958319 18115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "da0c2df8c87d435dbc869918eadf066b"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "8228a28e6a0ccfe82c4c955b040b0c59"
server_key_iv: "7c1225fbae47db3af6da9b4c985b3ecc"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.958771 18115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da0c2df8c87d435dbc869918eadf066b"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "8228a28e6a0ccfe82c4c955b040b0c59"
server_key_iv: "7c1225fbae47db3af6da9b4c985b3ecc"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.960064 18115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:53.960798 18129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.960975 18115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:53.961086 18115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "da0c2df8c87d435dbc869918eadf066b"
format_stamp: "Formatted at 2026-09-25 14:52:53 on dist-test-slave-z2s5"
server_key: "8228a28e6a0ccfe82c4c955b040b0c59"
server_key_iv: "7c1225fbae47db3af6da9b4c985b3ecc"
server_key_version: "encryptionkey@0"
I20260925 14:52:53.961159 18115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:53.973841 18115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:53.974159 18115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:53.974308 18115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:53.974592 18115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:53.974936 18115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:53.974989 18115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.975030 18115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:53.975057 18115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:53.980731 18115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41573
I20260925 14:52:53.980772 18242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41573 every 8 connection(s)
I20260925 14:52:53.981087 18115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:53.986588 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.981856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33419 (local address 127.0.98.62:38805)
0925 14:52:53.981936 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:53.981938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:53.982174 (+ 236us) server_negotiation.cc:407] Connection header received
0925 14:52:53.982817 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:53.982819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:53.982837 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:53.982900 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:53.983654 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.984251 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.985193 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.985392 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.985465 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.986262 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:53.986263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:53.986264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:53.986268 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:53.986290 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:53.986294 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:53.986341 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:53.986461 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:52:53.986527 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:52:53.986759 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:53.981850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38805 (local address 127.0.98.2:33419)
0925 14:52:53.982040 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:53.982082 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:53.982705 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:53.982954 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:53.982958 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:53.982966 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:53.983537 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.983544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.984334 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.984336 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.985098 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:53.985100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:53.985484 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:53.985485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:53.985593 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:53.986113 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:53.986113 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:53.986140 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:53.986163 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:53.986354 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:53.986358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:53.986361 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:53.986465 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 14:52:53.986561 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260925 14:52:53.987286 18243 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38805
I20260925 14:52:53.987388 18243 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:53.987562 18243 heartbeater.cc:511] Master 127.0.98.62:38805 requested a full tablet report, sending...
I20260925 14:52:53.987969 17924 ts_manager.cc:194] Registered new tserver with Master: da0c2df8c87d435dbc869918eadf066b (127.0.98.2:41573)
I20260925 14:52:53.987982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18115
I20260925 14:52:53.988095 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:53.988338 392 external_mini_cluster.cc:1428] Setting key a80288a44026e5c20666bf712e212673
I20260925 14:52:53.988435 17924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33419
I20260925 14:52:53.989296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38805
--builtin_ntp_servers=127.0.98.20:35653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:54.070683 18246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:54.070906 18246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:54.070942 18246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:54.072098 18246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:54.072160 18246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:54.072204 18246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:54.073567 18246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:54.073819 18246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:54.074110 18246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:54.079290 18252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.079301 18254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:54.079351 18251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:54.079497 18246 server_base.cc:1034] running on GCE node
I20260925 14:52:54.079730 18246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:54.079965 18246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:54.081131 18246 hybrid_clock.cc:648] HybridClock initialized: now 1790347974081135 us; error 65 us; skew 500 ppm
I20260925 14:52:54.082499 18246 webserver.cc:460] Webserver started at http://127.0.98.3:36229/ using document root <none> and password file <none>
I20260925 14:52:54.082823 18246 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:54.082873 18246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:54.082964 18246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:54.084147 18246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "e2843faeb5dc47988cbaa6ff6ec91f93"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-z2s5"
server_key: "b50f8f16d9b8b989d1b155aad2dd4524"
server_key_iv: "fadf10d0306533b74d6abef8878a519a"
server_key_version: "encryptionkey@0"
I20260925 14:52:54.084533 18246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e2843faeb5dc47988cbaa6ff6ec91f93"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-z2s5"
server_key: "b50f8f16d9b8b989d1b155aad2dd4524"
server_key_iv: "fadf10d0306533b74d6abef8878a519a"
server_key_version: "encryptionkey@0"
I20260925 14:52:54.085816 18246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:52:54.086663 18260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.086911 18246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:54.086993 18246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "e2843faeb5dc47988cbaa6ff6ec91f93"
format_stamp: "Formatted at 2026-09-25 14:52:54 on dist-test-slave-z2s5"
server_key: "b50f8f16d9b8b989d1b155aad2dd4524"
server_key_iv: "fadf10d0306533b74d6abef8878a519a"
server_key_version: "encryptionkey@0"
I20260925 14:52:54.087047 18246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:54.098703 18246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:54.098970 18246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:54.099077 18246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:54.099257 18246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:54.099521 18246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:54.099550 18246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.099571 18246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:54.099583 18246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:54.105489 18246 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38511
I20260925 14:52:54.105569 18373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38511 every 8 connection(s)
I20260925 14:52:54.105894 18246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:54.110791 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.106668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40701 (local address 127.0.98.62:38805)
0925 14:52:54.106747 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.106749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.106955 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:52:54.107428 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.107429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.107439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.107493 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.108119 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.108624 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.109587 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.109768 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.109818 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.110566 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.110567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.110567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:54.110571 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.110584 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:54.110588 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.110631 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.110712 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.110733 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:52:54.111021 18376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.106634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38805 (local address 127.0.98.3:40701)
0925 14:52:54.106837 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.106871 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.107346 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.107554 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.107557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.107565 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:54.108051 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.108055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.108706 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.108707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.109514 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.109517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.109844 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.109844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.109954 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.110440 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:54.110440 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:54.110469 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:54.110491 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:54.110643 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:54.110647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:54.110649 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.110704 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.110781 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260925 14:52:54.111516 18374 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38805
I20260925 14:52:54.111618 18374 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:54.111794 18374 heartbeater.cc:511] Master 127.0.98.62:38805 requested a full tablet report, sending...
I20260925 14:52:54.112175 17924 ts_manager.cc:194] Registered new tserver with Master: e2843faeb5dc47988cbaa6ff6ec91f93 (127.0.98.3:38511)
I20260925 14:52:54.112504 17924 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40701
I20260925 14:52:54.114141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18246
I20260925 14:52:54.114264 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:54.114534 392 external_mini_cluster.cc:1428] Setting key 9f25a53cf39293a3fb9b7f80f8f76f0e
I20260925 14:52:54.115370 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:54.119426 17963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.116764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.0.98.62:38805)
0925 14:52:54.116811 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.116813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.116896 (+ 83us) server_negotiation.cc:407] Connection header received
0925 14:52:54.116954 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.116955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.116962 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.117000 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:54.117122 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.117574 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.118275 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.118521 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.118570 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.119226 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:54.119227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:54.119227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:54.119231 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:54.119240 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:54.119244 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:54.119281 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.119316 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.119333 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:52:54.121042 17924 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:54.122624 17924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:54.129222 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.125210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.0.98.2:41573)
0925 14:52:54.125373 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.125377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.125427 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:52:54.125513 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.125516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.125531 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.125615 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.125796 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.126956 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.128039 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.128791 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.128873 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.128972 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.129087 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.129130 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":88,"threads_started":1}
I20260925 14:52:54.129228 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.124854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41573 (local address 127.0.0.1:36910)
0925 14:52:54.125373 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.125384 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.125435 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.125688 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.125689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.125690 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.125717 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.125721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.127051 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.127052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.127953 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.127955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.128895 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.128896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.128976 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.128990 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.129067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.129131 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":27,"spinlock_wait_cycles":384,"thread_start_us":91,"threads_started":1}
I20260925 14:52:54.129227 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.124873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38261 (local address 127.0.0.1:45476)
0925 14:52:54.125388 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.125399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.125440 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.125785 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.125786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.125788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.125839 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.125840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.126787 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.126788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.128061 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.128063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.128909 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.128910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.128976 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.128990 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.129067 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.129131 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":6,"thread_start_us":107,"threads_started":1}
I20260925 14:52:54.129500 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.125136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45476 (local address 127.0.98.1:38261)
0925 14:52:54.125555 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.125561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.125571 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:54.125621 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.125623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.125639 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.125719 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.125917 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.126697 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.128150 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.128840 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.128902 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.128972 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.129365 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.129420 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":85,"threads_started":1}
I20260925 14:52:54.130013 18177 tablet_service.cc:1467] Processing CreateTablet for tablet 232260315923406a8ca25e962dcd8f0f (DEFAULT_TABLE table=test-table [id=e9d7db6c58b64b8680881e304c7e04ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:54.130062 18046 tablet_service.cc:1467] Processing CreateTablet for tablet 232260315923406a8ca25e962dcd8f0f (DEFAULT_TABLE table=test-table [id=e9d7db6c58b64b8680881e304c7e04ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:54.130427 18046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232260315923406a8ca25e962dcd8f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:54.130427 18177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232260315923406a8ca25e962dcd8f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:54.130569 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.124854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38511 (local address 127.0.0.1:53014)
0925 14:52:54.125140 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.125182 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.125251 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.125647 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.125650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.125653 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.125717 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.125719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.127039 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.127040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.128105 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.128107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.130302 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.130304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.130372 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.130375 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.130450 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.130495 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":69,"threads_started":1}
I20260925 14:52:54.130975 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.125029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53014 (local address 127.0.98.3:38511)
0925 14:52:54.125214 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.125219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.125232 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:52:54.125343 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.125346 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.125362 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.125796 (+ 434us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.125851 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.126954 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.129016 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.130164 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.130768 (+ 604us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.130862 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.130883 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.130913 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":83,"threads_started":1}
I20260925 14:52:54.133778 18389 tablet_bootstrap.cc:492] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Bootstrap starting.
I20260925 14:52:54.134266 18308 tablet_service.cc:1467] Processing CreateTablet for tablet 232260315923406a8ca25e962dcd8f0f (DEFAULT_TABLE table=test-table [id=e9d7db6c58b64b8680881e304c7e04ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:54.134457 18389 tablet_bootstrap.cc:654] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:54.134620 18308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 232260315923406a8ca25e962dcd8f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:54.134632 18388 tablet_bootstrap.cc:492] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c: Bootstrap starting.
I20260925 14:52:54.134794 18389 log.cc:824] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:54.135317 18388 tablet_bootstrap.cc:654] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:54.135623 18388 log.cc:824] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:54.135654 18389 tablet_bootstrap.cc:492] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: No bootstrap required, opened a new log
I20260925 14:52:54.135718 18389 ts_tablet_manager.cc:1397] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:54.136404 18388 tablet_bootstrap.cc:492] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c: No bootstrap required, opened a new log
I20260925 14:52:54.136480 18388 ts_tablet_manager.cc:1397] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:54.138137 18392 tablet_bootstrap.cc:492] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Bootstrap starting.
I20260925 14:52:54.138114 18389 raft_consensus.cc:348] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.138288 18389 raft_consensus.cc:374] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:54.138331 18389 raft_consensus.cc:729] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0c2df8c87d435dbc869918eadf066b, State: Initialized, Role: FOLLOWER
I20260925 14:52:54.138484 18389 consensus_queue.cc:260] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.138764 18389 ts_tablet_manager.cc:1428] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:54.138844 18392 tablet_bootstrap.cc:654] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:54.138834 18243 heartbeater.cc:503] Master 127.0.98.62:38805 was elected leader, sending a full tablet report...
I20260925 14:52:54.139127 18392 log.cc:824] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:54.139087 18388 raft_consensus.cc:348] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.139236 18388 raft_consensus.cc:374] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:54.139281 18388 raft_consensus.cc:729] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 523bf94fa4b340d5a6f87e1554005e9c, State: Initialized, Role: FOLLOWER
I20260925 14:52:54.139403 18388 consensus_queue.cc:260] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.139674 18388 ts_tablet_manager.cc:1428] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:54.139715 18112 heartbeater.cc:503] Master 127.0.98.62:38805 was elected leader, sending a full tablet report...
I20260925 14:52:54.140285 18392 tablet_bootstrap.cc:492] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: No bootstrap required, opened a new log
I20260925 14:52:54.140355 18392 ts_tablet_manager.cc:1397] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:54.142189 18392 raft_consensus.cc:348] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.142309 18392 raft_consensus.cc:374] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:54.142330 18392 raft_consensus.cc:729] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2843faeb5dc47988cbaa6ff6ec91f93, State: Initialized, Role: FOLLOWER
I20260925 14:52:54.142465 18392 consensus_queue.cc:260] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.142728 18392 ts_tablet_manager.cc:1428] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 14:52:54.142766 18374 heartbeater.cc:503] Master 127.0.98.62:38805 was elected leader, sending a full tablet report...
W20260925 14:52:54.231734 18244 tablet.cc:2369] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:54.273115 18395 raft_consensus.cc:482] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:54.273236 18395 raft_consensus.cc:504] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.273551 18395 leader_election.cc:288] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2843faeb5dc47988cbaa6ff6ec91f93 (127.0.98.3:38511), da0c2df8c87d435dbc869918eadf066b (127.0.98.2:41573)
I20260925 14:52:54.276526 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.273838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54007 (local address 127.0.98.3:38511)
0925 14:52:54.273909 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.273912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.273951 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:52:54.273991 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.273992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.274000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.274063 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.274251 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.274779 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.275739 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.276164 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.276252 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.276262 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.276425 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.276461 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:52:54.276535 18114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.273768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38511 (local address 127.0.98.1:54007)
0925 14:52:54.273836 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.273849 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.273899 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.274096 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.274096 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.274098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.274179 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.274181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.274865 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.274866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.275635 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.275638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.276255 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.276256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.276340 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.276343 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.276406 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.276449 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:52:54.276859 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.273913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39705 (local address 127.0.98.2:41573)
0925 14:52:54.273957 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.273959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.274241 (+ 282us) server_negotiation.cc:407] Connection header received
0925 14:52:54.274268 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.274268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.274276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.274343 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:54.274514 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.274969 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.276002 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.276397 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.276516 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.276527 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.276734 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.276784 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:52:54.276855 18328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232260315923406a8ca25e962dcd8f0f" candidate_uuid: "523bf94fa4b340d5a6f87e1554005e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" is_pre_election: true
I20260925 14:52:54.277032 18328 raft_consensus.cc:2454] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523bf94fa4b340d5a6f87e1554005e9c in term 0.
I20260925 14:52:54.277094 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.273772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41573 (local address 127.0.98.1:39705)
0925 14:52:54.274013 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.274020 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:54.274086 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:54.274344 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:54.274345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:54.274347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:54.274432 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.274434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.275053 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.275054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.275899 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:54.275902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.276536 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.276537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:54.276636 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.276640 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:54.276964 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 14:52:54.277010 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":38,"thread_start_us":106,"threads_started":1}
I20260925 14:52:54.277297 18001 leader_election.cc:302] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523bf94fa4b340d5a6f87e1554005e9c, e2843faeb5dc47988cbaa6ff6ec91f93; no voters:
I20260925 14:52:54.277441 18395 raft_consensus.cc:2781] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:54.277436 18197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232260315923406a8ca25e962dcd8f0f" candidate_uuid: "523bf94fa4b340d5a6f87e1554005e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0c2df8c87d435dbc869918eadf066b" is_pre_election: true
I20260925 14:52:54.277508 18395 raft_consensus.cc:482] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:54.277536 18395 raft_consensus.cc:3037] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:54.277588 18197 raft_consensus.cc:2454] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 523bf94fa4b340d5a6f87e1554005e9c in term 0.
I20260925 14:52:54.278304 18395 raft_consensus.cc:504] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.278481 18395 leader_election.cc:288] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [CANDIDATE]: Term 1 election: Requested vote from peers e2843faeb5dc47988cbaa6ff6ec91f93 (127.0.98.3:38511), da0c2df8c87d435dbc869918eadf066b (127.0.98.2:41573)
I20260925 14:52:54.278633 18197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232260315923406a8ca25e962dcd8f0f" candidate_uuid: "523bf94fa4b340d5a6f87e1554005e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da0c2df8c87d435dbc869918eadf066b"
I20260925 14:52:54.278648 18328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "232260315923406a8ca25e962dcd8f0f" candidate_uuid: "523bf94fa4b340d5a6f87e1554005e9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93"
I20260925 14:52:54.278721 18328 raft_consensus.cc:3037] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:54.278712 18197 raft_consensus.cc:3037] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:54.279620 18328 raft_consensus.cc:2454] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523bf94fa4b340d5a6f87e1554005e9c in term 1.
I20260925 14:52:54.279662 18197 raft_consensus.cc:2454] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 523bf94fa4b340d5a6f87e1554005e9c in term 1.
I20260925 14:52:54.279824 18001 leader_election.cc:302] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 523bf94fa4b340d5a6f87e1554005e9c, e2843faeb5dc47988cbaa6ff6ec91f93; no voters:
I20260925 14:52:54.279953 18395 raft_consensus.cc:2781] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:54.280117 18395 raft_consensus.cc:686] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [term 1 LEADER]: Becoming Leader. State: Replica: 523bf94fa4b340d5a6f87e1554005e9c, State: Running, Role: LEADER
I20260925 14:52:54.280237 18395 consensus_queue.cc:237] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } }
I20260925 14:52:54.281075 17924 catalog_manager.cc:5495] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c reported cstate change: term changed from 0 to 1, leader changed from <none> to 523bf94fa4b340d5a6f87e1554005e9c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "523bf94fa4b340d5a6f87e1554005e9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "523bf94fa4b340d5a6f87e1554005e9c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38261 } health_report { overall_health: HEALTHY } } }
I20260925 14:52:54.310081 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.307630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.0.98.1:38261)
0925 14:52:54.307686 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:54.307689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:54.307703 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:54.307756 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:54.307757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:54.307778 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:54.307861 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:54.307974 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.308459 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.309269 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:54.309466 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:54.309543 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:54.309706 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:54.309813 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:54.309912 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:54.309931 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:52:54.309952 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:52:54.311913 18197 raft_consensus.cc:1261] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 1 FOLLOWER]: Refusing update from remote peer 523bf94fa4b340d5a6f87e1554005e9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:54.311955 18328 raft_consensus.cc:1261] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 1 FOLLOWER]: Refusing update from remote peer 523bf94fa4b340d5a6f87e1554005e9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:54.312304 18398 consensus_queue.cc:1035] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "da0c2df8c87d435dbc869918eadf066b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41573 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:54.312429 18395 consensus_queue.cc:1035] T 232260315923406a8ca25e962dcd8f0f P 523bf94fa4b340d5a6f87e1554005e9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2843faeb5dc47988cbaa6ff6ec91f93" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:54.315585 18403 mvcc.cc:205] Tried to move back new op lower bound from 7333265302778826752 to 7333265302652276736. Current Snapshot: MvccSnapshot[applied={T|T < 7333265302778826752}]
I20260925 14:52:54.317256 17922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:54.317348 17922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:54.318164 17922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c6d454741914b6fa4c5a121f8431cc5: Sending DeleteTablet for 3 replicas of tablet 232260315923406a8ca25e962dcd8f0f
I20260925 14:52:54.318526 18308 tablet_service.cc:1514] Processing DeleteTablet for tablet 232260315923406a8ca25e962dcd8f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:54 UTC) from {username='slave'} at 127.0.0.1:53014
I20260925 14:52:54.318590 18046 tablet_service.cc:1514] Processing DeleteTablet for tablet 232260315923406a8ca25e962dcd8f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:54 UTC) from {username='slave'} at 127.0.0.1:45476
I20260925 14:52:54.318770 18177 tablet_service.cc:1514] Processing DeleteTablet for tablet 232260315923406a8ca25e962dcd8f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:54 UTC) from {username='slave'} at 127.0.0.1:36910
I20260925 14:52:54.318818 18411 tablet_replica.cc:330] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: stopping tablet replica
I20260925 14:52:54.318879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17984
I20260925 14:52:54.318918 18411 raft_consensus.cc:2229] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:54.319370 18413 tablet_replica.cc:330] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: stopping tablet replica
I20260925 14:52:54.319454 18413 raft_consensus.cc:2229] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:54.319547 18413 raft_consensus.cc:2258] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:54.320161 18413 ts_tablet_manager.cc:1905] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:54.320755 18411 raft_consensus.cc:2258] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:54.321233 18411 ts_tablet_manager.cc:1905] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:54.321590 18413 ts_tablet_manager.cc:1918] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:54.321669 18413 log.cc:1196] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/232260315923406a8ca25e962dcd8f0f
I20260925 14:52:54.321951 18413 ts_tablet_manager.cc:1939] T 232260315923406a8ca25e962dcd8f0f P da0c2df8c87d435dbc869918eadf066b: Deleting consensus metadata
I20260925 14:52:54.322487 17911 catalog_manager.cc:4841] TS da0c2df8c87d435dbc869918eadf066b (127.0.98.2:41573): tablet 232260315923406a8ca25e962dcd8f0f (table test-table [id=e9d7db6c58b64b8680881e304c7e04ce]) successfully deleted
I20260925 14:52:54.323191 18411 ts_tablet_manager.cc:1918] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:54.323271 18411 log.cc:1196] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/232260315923406a8ca25e962dcd8f0f
I20260925 14:52:54.323503 18411 ts_tablet_manager.cc:1939] T 232260315923406a8ca25e962dcd8f0f P e2843faeb5dc47988cbaa6ff6ec91f93: Deleting consensus metadata
I20260925 14:52:54.324244 17910 catalog_manager.cc:4841] TS e2843faeb5dc47988cbaa6ff6ec91f93 (127.0.98.3:38511): tablet 232260315923406a8ca25e962dcd8f0f (table test-table [id=e9d7db6c58b64b8680881e304c7e04ce]) successfully deleted
W20260925 14:52:54.325670 17909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38261 (error 108)
I20260925 14:52:54.326197 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:54.326050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38261 (local address 127.0.0.1:45500)
0925 14:52:54.326113 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:54.326158 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:52:54.326192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18115
W20260925 14:52:54.326320 17909 catalog_manager.cc:4584] TS 523bf94fa4b340d5a6f87e1554005e9c (127.0.98.1:38261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 232260315923406a8ca25e962dcd8f0f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38261: connect: Connection refused (error 111)
I20260925 14:52:54.331696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18246
I20260925 14:52:54.336193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 17893
2026-09-25T14:52:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (780 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:54 dist-test-slave-z2s5 krb5kdc[18417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:54 dist-test-slave-z2s5 krb5kdc[18417](info): set up 2 sockets
Sep 25 14:52:54 dist-test-slave-z2s5 krb5kdc[18417](info): commencing operation
krb5kdc: starting...
W20260925 14:52:55.378327 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:52:55.509677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:55.593822 18430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.594051 18430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.594089 18430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.595338 18430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:55.595412 18430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.595435 18430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.595458 18430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:55.595487 18430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:55.597071 18430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:55.597354 18430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.597584 18430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:55.602480 18435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.602593 18438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.602519 18436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.602782 18430 server_base.cc:1034] running on GCE node
I20260925 14:52:55.602962 18430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.603197 18430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.604336 18430 hybrid_clock.cc:648] HybridClock initialized: now 1790347975604307 us; error 45 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.606750 18430 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:55.607227 18430 webserver.cc:460] Webserver started at http://127.0.98.62:32875/ using document root <none> and password file <none>
I20260925 14:52:55.607462 18430 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.607532 18430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:55.607683 18430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:55.608754 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "9e17d5d909694e3cac86e86c6ec3d98c"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "d00c06aff6174368f91b1f48dc50052d"
server_key_iv: "ed9a5f2c6773e20a4969585f65b1eaf5"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.609109 18430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e17d5d909694e3cac86e86c6ec3d98c"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "d00c06aff6174368f91b1f48dc50052d"
server_key_iv: "ed9a5f2c6773e20a4969585f65b1eaf5"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.610491 18430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:52:55.611277 18445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.611459 18430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:55.611559 18430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "9e17d5d909694e3cac86e86c6ec3d98c"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "d00c06aff6174368f91b1f48dc50052d"
server_key_iv: "ed9a5f2c6773e20a4969585f65b1eaf5"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.611685 18430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.632550 18430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.634730 18430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.634894 18430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.638873 18430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39195
I20260925 14:52:55.638932 18497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39195 every 8 connection(s)
I20260925 14:52:55.639292 18430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:55.640151 18498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:55.642481 18498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: Bootstrap starting.
I20260925 14:52:55.643049 18498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:55.643350 18498 log.cc:824] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:55.644054 18498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: No bootstrap required, opened a new log
I20260925 14:52:55.644466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18430
I20260925 14:52:55.644589 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:55.644806 392 external_mini_cluster.cc:1428] Setting key fa262c85dc3d6942d3313562f67a2f07
I20260925 14:52:55.646202 18498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39195 } }
I20260925 14:52:55.646366 18498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:55.646427 18498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e17d5d909694e3cac86e86c6ec3d98c, State: Initialized, Role: FOLLOWER
I20260925 14:52:55.646607 18498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39195 } }
I20260925 14:52:55.646705 18498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:55.646750 18498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:55.646800 18498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:55.647392 18498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39195 } }
I20260925 14:52:55.647498 18498 leader_election.cc:302] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e17d5d909694e3cac86e86c6ec3d98c; no voters:
I20260925 14:52:55.647657 18498 leader_election.cc:288] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:55.647730 18503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:55.647883 18503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [term 1 LEADER]: Becoming Leader. State: Replica: 9e17d5d909694e3cac86e86c6ec3d98c, State: Running, Role: LEADER
I20260925 14:52:55.647903 18498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:55.647966 18503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39195 } }
I20260925 14:52:55.648311 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e17d5d909694e3cac86e86c6ec3d98c. Latest consensus state: current_term: 1 leader_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39195 } } }
I20260925 14:52:55.648406 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:55.648321 18504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e17d5d909694e3cac86e86c6ec3d98c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39195 } } }
I20260925 14:52:55.648497 18504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c [sys.catalog]: This master's current role is: LEADER
W20260925 14:52:55.649765 18519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:52:55.649822 18519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:52:55.649873 18510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:55.650048 18510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:55.651628 18510 catalog_manager.cc:1347] Generated new cluster ID: c957c47904334c0daee9fdb0a4e4ee09
I20260925 14:52:55.651686 18510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:55.657269 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.645346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.0.98.62:39195)
0925 14:52:55.645520 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.645528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.645553 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:52:55.646338 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.646346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.646445 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.646616 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.647407 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.648929 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.649940 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.650571 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.650696 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.653490 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.653494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.653494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.653516 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.655584 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.655973 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.655977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.655978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.656095 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.656311 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.656311 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.656312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.656633 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.656783 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.656909 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.657047 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260925 14:52:55.662509 18510 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:55.663009 18510 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:55.666354 18510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: Generated new TSK 0
I20260925 14:52:55.666538 18510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:52:55.695008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39195
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:55.778632 18524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.778851 18524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.778877 18524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.780170 18524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.780218 18524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.780246 18524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:55.781792 18524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:55.782074 18524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.782335 18524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:52:55.787370 18524 server_base.cc:1034] running on GCE node
W20260925 14:52:55.787351 18530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.787336 18532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.787326 18529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.787802 18524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.788013 18524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.789191 18524 hybrid_clock.cc:648] HybridClock initialized: now 1790347975789179 us; error 40 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.791827 18524 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:55.792265 18524 webserver.cc:460] Webserver started at http://127.0.98.1:37395/ using document root <none> and password file <none>
I20260925 14:52:55.792491 18524 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.792562 18524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:55.792711 18524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:55.793780 18524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "48c541129ddd4c0a9fcbf5f3c6a18510"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "8962fc2297e85d30326a1981b7f89406"
server_key_iv: "a5a6621e79a2d040ea8fead6205ede74"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.794153 18524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "48c541129ddd4c0a9fcbf5f3c6a18510"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "8962fc2297e85d30326a1981b7f89406"
server_key_iv: "a5a6621e79a2d040ea8fead6205ede74"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.795574 18524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260925 14:52:55.796334 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.796533 18524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:55.796631 18524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "48c541129ddd4c0a9fcbf5f3c6a18510"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "8962fc2297e85d30326a1981b7f89406"
server_key_iv: "a5a6621e79a2d040ea8fead6205ede74"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.796717 18524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.814474 18524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.816953 18524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.817121 18524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.817375 18524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:55.817690 18524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:55.817751 18524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.817799 18524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:55.817822 18524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.823943 18524 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42975
I20260925 14:52:55.823988 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42975 every 8 connection(s)
I20260925 14:52:55.824345 18524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:55.830209 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18524
I20260925 14:52:55.830307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:55.830523 392 external_mini_cluster.cc:1428] Setting key a348d608bdc2771a184033ab9dd2be2c
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:55.834574 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.825153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53993 (local address 127.0.98.62:39195)
0925 14:52:55.825226 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.825228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.825459 (+ 231us) server_negotiation.cc:407] Connection header received
0925 14:52:55.826432 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.826433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.826445 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.826516 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.827099 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.827596 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.828603 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.828796 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.828846 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.831438 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:55.831439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:55.831439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:55.831445 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:55.833137 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.833524 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.833526 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.833526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.833590 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:55.833848 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:55.833849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:55.833849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:55.833998 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:55.834067 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:55.834311 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 14:52:55.834426 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:52:55.834607 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.825127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39195 (local address 127.0.98.1:53993)
0925 14:52:55.825321 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.825364 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.826305 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.826579 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.826583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.826831 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:55.827023 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.827026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.827689 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.827690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.828524 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.828527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.828871 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.828872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.828980 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.829462 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:55.829470 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:55.831326 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:55.833227 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.833234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.833236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.833449 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.833657 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:55.833658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:55.833658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:55.833781 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:55.834063 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:55.834067 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:55.834182 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 14:52:55.834266 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:52:55.834426 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":113,"threads_started":1}
I20260925 14:52:55.835844 18653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39195
I20260925 14:52:55.835945 18653 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:55.836146 18653 heartbeater.cc:511] Master 127.0.98.62:39195 requested a full tablet report, sending...
I20260925 14:52:55.836683 18462 ts_manager.cc:194] Registered new tserver with Master: 48c541129ddd4c0a9fcbf5f3c6a18510 (127.0.98.1:42975)
I20260925 14:52:55.837311 18462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:52:55.856873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39195
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:55.938125 18658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:55.938388 18658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:55.938450 18658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:55.939628 18658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:55.939718 18658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:55.939766 18658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:55.941255 18658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:55.941550 18658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:55.941787 18658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:55.946766 18663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.946904 18658 server_base.cc:1034] running on GCE node
W20260925 14:52:55.946767 18664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:55.946991 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:55.947218 18658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:55.947433 18658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:55.948684 18658 hybrid_clock.cc:648] HybridClock initialized: now 1790347975948666 us; error 29 us; skew 500 ppm
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:55.951647 18658 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:55.952102 18658 webserver.cc:460] Webserver started at http://127.0.98.2:36329/ using document root <none> and password file <none>
I20260925 14:52:55.952333 18658 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:55.952379 18658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:55.952546 18658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:55.953624 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "f53774033e9e4fd5abe508e0708b2b11"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "ecc902a31f8955ed1f45cc91aee67a44"
server_key_iv: "4fd57225d6aee0c0857626538e4cf739"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.953986 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f53774033e9e4fd5abe508e0708b2b11"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "ecc902a31f8955ed1f45cc91aee67a44"
server_key_iv: "4fd57225d6aee0c0857626538e4cf739"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.955381 18658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:52:55.956120 18673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.956347 18658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:52:55.956453 18658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "f53774033e9e4fd5abe508e0708b2b11"
format_stamp: "Formatted at 2026-09-25 14:52:55 on dist-test-slave-z2s5"
server_key: "ecc902a31f8955ed1f45cc91aee67a44"
server_key_iv: "4fd57225d6aee0c0857626538e4cf739"
server_key_version: "encryptionkey@0"
I20260925 14:52:55.956529 18658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:55.984866 18658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:55.987017 18658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:55.987181 18658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:55.987447 18658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:55.987757 18658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:55.987790 18658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.987813 18658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:55.987860 18658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:55.993659 18658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43203
I20260925 14:52:55.993742 18786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43203 every 8 connection(s)
I20260925 14:52:55.994064 18658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:52:55 dist-test-slave-z2s5 krb5kdc[18417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347975, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:56.001830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18658
I20260925 14:52:56.001959 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:56.002140 392 external_mini_cluster.cc:1428] Setting key c6e3288935a37fc7356fe6bb84cc506e
I20260925 14:52:56.004042 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.994877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52861 (local address 127.0.98.62:39195)
0925 14:52:55.994944 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:55.994948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:55.995202 (+ 254us) server_negotiation.cc:407] Connection header received
0925 14:52:55.996105 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:55.996106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:55.996118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:55.996195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:55.996773 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.997269 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.998242 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.998490 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.998545 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.000838 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.000839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.000839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:56.000846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.002555 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.003052 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.003053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.003053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.003113 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.003381 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.003382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.003383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.003502 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.003581 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.003868 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.003948 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:52:56.004256 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:55.994859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39195 (local address 127.0.98.2:52861)
0925 14:52:55.995034 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:55.995079 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:55.996005 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:55.996244 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:55.996247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:55.996506 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:55.996697 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.996701 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.997349 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.997350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.998166 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:55.998169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:55.998558 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:55.998559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:55.998662 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:55.999184 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:55.999192 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:56.000753 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:56.002642 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:56.002650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:56.002652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:56.002957 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:56.003186 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:56.003188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:56.003188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:56.003311 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:56.003571 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:56.003578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:56.003731 (+ 153us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.003845 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.004010 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260925 14:52:56.004765 18787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39195
I20260925 14:52:56.004876 18787 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.005057 18787 heartbeater.cc:511] Master 127.0.98.62:39195 requested a full tablet report, sending...
I20260925 14:52:56.005439 18462 ts_manager.cc:194] Registered new tserver with Master: f53774033e9e4fd5abe508e0708b2b11 (127.0.98.2:43203)
I20260925 14:52:56.005981 18462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52861
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:52:56.028720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39195
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:56.109790 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:56.110003 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:56.110025 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:56.111184 18792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:56.111222 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:56.111256 18792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:56.112768 18792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:56.113019 18792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:56.113271 18792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:56.118455 18798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.118417 18797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.118417 18800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:56.118854 18792 server_base.cc:1034] running on GCE node
I20260925 14:52:56.119103 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:56.119370 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:56.120533 18792 hybrid_clock.cc:648] HybridClock initialized: now 1790347976120528 us; error 40 us; skew 500 ppm
Sep 25 14:52:56 dist-test-slave-z2s5 krb5kdc[18417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:56.123030 18792 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:56.123528 18792 webserver.cc:460] Webserver started at http://127.0.98.3:36789/ using document root <none> and password file <none>
I20260925 14:52:56.123759 18792 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:56.123826 18792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:56.123978 18792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:56.125026 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "48f343a8ba144f2d926854fb5d26931d"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "24035ab30be0654f1e461dbaedae3611"
server_key_iv: "7592b67e2d6a720ccfbdf15dd28d2108"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.125394 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "48f343a8ba144f2d926854fb5d26931d"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "24035ab30be0654f1e461dbaedae3611"
server_key_iv: "7592b67e2d6a720ccfbdf15dd28d2108"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.126745 18792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:52:56.127418 18807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.127583 18792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:52:56.127686 18792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "48f343a8ba144f2d926854fb5d26931d"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "24035ab30be0654f1e461dbaedae3611"
server_key_iv: "7592b67e2d6a720ccfbdf15dd28d2108"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.127760 18792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:56.144878 18792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:56.147385 18792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:56.147562 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:56.147816 18792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:56.148170 18792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:56.148234 18792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.148290 18792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:56.148317 18792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.154453 18792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40703
I20260925 14:52:56.154484 18920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40703 every 8 connection(s)
I20260925 14:52:56.154816 18792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:52:56 dist-test-slave-z2s5 krb5kdc[18417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:56.163834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18792
I20260925 14:52:56.163941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:56.164098 392 external_mini_cluster.cc:1428] Setting key 0e29709921ca4f65346c3790c7841c3b
I20260925 14:52:56.164466 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.155492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35553 (local address 127.0.98.62:39195)
0925 14:52:56.155544 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.155545 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.155780 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:52:56.156804 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.156805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.156814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.156876 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.157509 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.158010 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.159081 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.159314 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.159370 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.161536 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.161538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.161538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:56.161544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.163048 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.163434 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.163434 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.163435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.163483 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.163768 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.163769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.163770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.163901 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.163984 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.164290 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.164369 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:52:56.164685 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.155471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39195 (local address 127.0.98.3:35553)
0925 14:52:56.155641 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.155682 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.156707 (+ 1025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.156948 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.156951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.157224 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:56.157420 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.157428 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.158099 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.158100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.159008 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.159010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.159389 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.159390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.159489 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.159992 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:56.160001 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:56.161455 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:56.163139 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:56.163145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:56.163146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:56.163364 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:56.163558 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:56.163560 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:56.163560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:56.163684 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:56.163975 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:56.163981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:56.164122 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.164255 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.164421 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260925 14:52:56.165150 18921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39195
I20260925 14:52:56.165261 18921 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.165437 18921 heartbeater.cc:511] Master 127.0.98.62:39195 requested a full tablet report, sending...
I20260925 14:52:56.165761 18462 ts_manager.cc:194] Registered new tserver with Master: 48f343a8ba144f2d926854fb5d26931d (127.0.98.3:40703)
I20260925 14:52:56.166090 18462 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35553
I20260925 14:52:56.175990 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:56.183609 18502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.177321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.0.98.62:39195)
0925 14:52:56.177382 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.177384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.177497 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:52:56.177586 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.177587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.177598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.177716 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.178268 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.178827 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.179574 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.179786 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.179866 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.181133 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.181134 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.181134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:56.181142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.182344 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.182769 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.182770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.182771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.182831 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:56.183051 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:56.183053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:56.183053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:56.183202 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.183322 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.183442 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.183510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:52:56.185127 18462 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:56.186757 18462 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:56.193284 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.189055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42975 (local address 127.0.0.1:40616)
0925 14:52:56.189279 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.189323 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.189386 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.189757 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.189761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.189763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.189841 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.189843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.190894 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.190896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.192085 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.192088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.192963 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.192964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.193061 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.193076 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.193150 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.193198 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":60,"threads_started":1}
I20260925 14:52:56.193325 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.189425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.0.98.1:42975)
0925 14:52:56.189569 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.189573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.189584 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:52:56.189624 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.189626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.189643 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.189745 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.189927 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.190785 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.192163 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.192857 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.192937 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.192997 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.193199 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.193244 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260925 14:52:56.193436 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.189108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.0.1:32804)
0925 14:52:56.189566 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.189576 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.189625 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.189911 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.189912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.189916 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.189974 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.189977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.191314 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.191315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.192318 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.192320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.193221 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.193222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.193285 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.193290 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.193320 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.193366 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"mutex_wait_us":9,"thread_start_us":94,"threads_started":1}
I20260925 14:52:56.193713 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.189243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32804 (local address 127.0.98.3:40703)
0925 14:52:56.189750 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.189754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.189763 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:52:56.189799 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.189801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.189816 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.190047 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.190123 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.191196 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.192396 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.193102 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.193507 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.193556 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.193576 (+ 20us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.193608 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":79,"threads_started":1}
I20260925 14:52:56.194438 18587 tablet_service.cc:1467] Processing CreateTablet for tablet 2f446e7d2d94481a9492c38d7e65789f (DEFAULT_TABLE table=test-table [id=3ea7d787e6b64c12a681ca0525b1b272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:56.194437 18855 tablet_service.cc:1467] Processing CreateTablet for tablet 2f446e7d2d94481a9492c38d7e65789f (DEFAULT_TABLE table=test-table [id=3ea7d787e6b64c12a681ca0525b1b272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:56.194561 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.189261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43203 (local address 127.0.0.1:51718)
0925 14:52:56.189583 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.189593 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.189649 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.189911 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.189912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.189914 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.189974 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.189976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.192387 (+ 2411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.192388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.193267 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.193270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.194288 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.194289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.194350 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.194354 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.194457 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.194496 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260925 14:52:56.194589 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.189373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51718 (local address 127.0.98.2:43203)
0925 14:52:56.189548 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.189554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.189657 (+ 103us) server_negotiation.cc:407] Connection header received
0925 14:52:56.189728 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.189731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.189746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.189825 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.190054 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.191207 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.193361 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.194183 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.194283 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.194338 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.194475 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.194515 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260925 14:52:56.194794 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f446e7d2d94481a9492c38d7e65789f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.194794 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f446e7d2d94481a9492c38d7e65789f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.195444 18721 tablet_service.cc:1467] Processing CreateTablet for tablet 2f446e7d2d94481a9492c38d7e65789f (DEFAULT_TABLE table=test-table [id=3ea7d787e6b64c12a681ca0525b1b272]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:56.195758 18721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f446e7d2d94481a9492c38d7e65789f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.198225 18938 tablet_bootstrap.cc:492] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Bootstrap starting.
I20260925 14:52:56.198376 18939 tablet_bootstrap.cc:492] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Bootstrap starting.
I20260925 14:52:56.198601 18940 tablet_bootstrap.cc:492] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: Bootstrap starting.
I20260925 14:52:56.199128 18938 tablet_bootstrap.cc:654] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.199131 18939 tablet_bootstrap.cc:654] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.199204 18940 tablet_bootstrap.cc:654] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.199692 18938 log.cc:824] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:56.199810 18939 log.cc:824] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:56.199947 18940 log.cc:824] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:56.200556 18938 tablet_bootstrap.cc:492] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: No bootstrap required, opened a new log
I20260925 14:52:56.200610 18940 tablet_bootstrap.cc:492] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: No bootstrap required, opened a new log
I20260925 14:52:56.200634 18938 ts_tablet_manager.cc:1397] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:56.200660 18940 ts_tablet_manager.cc:1397] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:52:56.202790 18939 tablet_bootstrap.cc:492] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: No bootstrap required, opened a new log
I20260925 14:52:56.202644 18938 raft_consensus.cc:348] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.202644 18940 raft_consensus.cc:348] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.202854 18940 raft_consensus.cc:374] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.202854 18938 raft_consensus.cc:374] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.202881 18939 ts_tablet_manager.cc:1397] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:52:56.202901 18940 raft_consensus.cc:729] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48c541129ddd4c0a9fcbf5f3c6a18510, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.202901 18938 raft_consensus.cc:729] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53774033e9e4fd5abe508e0708b2b11, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.203016 18938 consensus_queue.cc:260] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.203016 18940 consensus_queue.cc:260] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.203300 18938 ts_tablet_manager.cc:1428] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:56.203300 18940 ts_tablet_manager.cc:1428] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:56.203424 18653 heartbeater.cc:503] Master 127.0.98.62:39195 was elected leader, sending a full tablet report...
I20260925 14:52:56.203481 18787 heartbeater.cc:503] Master 127.0.98.62:39195 was elected leader, sending a full tablet report...
I20260925 14:52:56.205018 18939 raft_consensus.cc:348] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.205204 18939 raft_consensus.cc:374] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.205250 18939 raft_consensus.cc:729] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f343a8ba144f2d926854fb5d26931d, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.205365 18939 consensus_queue.cc:260] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.205644 18939 ts_tablet_manager.cc:1428] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:52:56.205680 18921 heartbeater.cc:503] Master 127.0.98.62:39195 was elected leader, sending a full tablet report...
W20260925 14:52:56.244715 18788 tablet.cc:2369] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:56.325129 18654 tablet.cc:2369] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:56.405356 18922 tablet.cc:2369] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:56.454205 18945 raft_consensus.cc:482] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:56.454332 18945 raft_consensus.cc:504] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.454751 18945 leader_election.cc:288] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 48c541129ddd4c0a9fcbf5f3c6a18510 (127.0.98.1:42975), 48f343a8ba144f2d926854fb5d26931d (127.0.98.3:40703)
I20260925 14:52:56.458734 18937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.455093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59157 (local address 127.0.98.1:42975)
0925 14:52:56.455140 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.455142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.455156 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:56.455193 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.455193 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.455206 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.455287 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.455458 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.456281 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.457786 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.458336 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.458440 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.458451 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.458626 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.458663 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:52:56.458734 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.455119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46009 (local address 127.0.98.3:40703)
0925 14:52:56.455171 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.455173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.455380 (+ 207us) server_negotiation.cc:407] Connection header received
0925 14:52:56.455407 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.455408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.455434 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.455493 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:56.455703 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.456384 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.457787 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.458356 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.458440 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.458451 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.458619 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.458664 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:52:56.458750 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.454977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42975 (local address 127.0.98.2:59157)
0925 14:52:56.455046 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.455060 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.455129 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.455301 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.455303 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.455305 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.455380 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.455382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.456361 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.456362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.457679 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.457683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.458450 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.458451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.458538 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.458539 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.458599 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.458648 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":54}
I20260925 14:52:56.458756 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.454977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40703 (local address 127.0.98.2:46009)
0925 14:52:56.455294 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.455303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.455358 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.455530 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.455531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.455533 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:56.455620 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.455621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.456466 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.456467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.457679 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.457683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.458450 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.458451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.458535 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.458539 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.458599 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.458648 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":96,"thread_start_us":151,"threads_started":1}
I20260925 14:52:56.459167 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f446e7d2d94481a9492c38d7e65789f" candidate_uuid: "f53774033e9e4fd5abe508e0708b2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" is_pre_election: true
I20260925 14:52:56.459375 18607 raft_consensus.cc:2454] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53774033e9e4fd5abe508e0708b2b11 in term 0.
I20260925 14:52:56.459476 18875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f446e7d2d94481a9492c38d7e65789f" candidate_uuid: "f53774033e9e4fd5abe508e0708b2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f343a8ba144f2d926854fb5d26931d" is_pre_election: true
I20260925 14:52:56.459610 18677 leader_election.cc:302] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c541129ddd4c0a9fcbf5f3c6a18510, f53774033e9e4fd5abe508e0708b2b11; no voters:
I20260925 14:52:56.459671 18875 raft_consensus.cc:2454] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53774033e9e4fd5abe508e0708b2b11 in term 0.
I20260925 14:52:56.459729 18945 raft_consensus.cc:2781] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:56.459775 18945 raft_consensus.cc:482] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:56.459801 18945 raft_consensus.cc:3037] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:56.460546 18945 raft_consensus.cc:504] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.460697 18945 leader_election.cc:288] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [CANDIDATE]: Term 1 election: Requested vote from peers 48c541129ddd4c0a9fcbf5f3c6a18510 (127.0.98.1:42975), 48f343a8ba144f2d926854fb5d26931d (127.0.98.3:40703)
I20260925 14:52:56.460855 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f446e7d2d94481a9492c38d7e65789f" candidate_uuid: "f53774033e9e4fd5abe508e0708b2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510"
I20260925 14:52:56.460937 18607 raft_consensus.cc:3037] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:56.461665 18607 raft_consensus.cc:2454] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53774033e9e4fd5abe508e0708b2b11 in term 1.
I20260925 14:52:56.461783 18875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f446e7d2d94481a9492c38d7e65789f" candidate_uuid: "f53774033e9e4fd5abe508e0708b2b11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48f343a8ba144f2d926854fb5d26931d"
I20260925 14:52:56.461818 18677 leader_election.cc:302] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48c541129ddd4c0a9fcbf5f3c6a18510, f53774033e9e4fd5abe508e0708b2b11; no voters:
I20260925 14:52:56.461869 18875 raft_consensus.cc:3037] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:56.461949 18945 raft_consensus.cc:2781] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:56.462049 18945 raft_consensus.cc:686] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 1 LEADER]: Becoming Leader. State: Replica: f53774033e9e4fd5abe508e0708b2b11, State: Running, Role: LEADER
I20260925 14:52:56.462157 18945 consensus_queue.cc:237] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } }
I20260925 14:52:56.462792 18875 raft_consensus.cc:2454] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53774033e9e4fd5abe508e0708b2b11 in term 1.
I20260925 14:52:56.463174 18462 catalog_manager.cc:5495] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53774033e9e4fd5abe508e0708b2b11 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "f53774033e9e4fd5abe508e0708b2b11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53774033e9e4fd5abe508e0708b2b11" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:56.481160 18936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.478761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.0.98.2:43203)
0925 14:52:56.478825 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.478827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.478851 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:52:56.478887 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.478887 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.478907 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.478966 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:56.479089 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.479565 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.480341 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.480535 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.480641 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.480790 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:56.480884 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:56.480955 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.480996 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.481006 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:52:56.483157 18875 raft_consensus.cc:1261] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 1 FOLLOWER]: Refusing update from remote peer f53774033e9e4fd5abe508e0708b2b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:56.483170 18607 raft_consensus.cc:1261] T 2f446e7d2d94481a9492c38d7e65789f P 48c541129ddd4c0a9fcbf5f3c6a18510 [term 1 FOLLOWER]: Refusing update from remote peer f53774033e9e4fd5abe508e0708b2b11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:56.483521 18945 consensus_queue.cc:1035] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48f343a8ba144f2d926854fb5d26931d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:56.483683 18948 consensus_queue.cc:1035] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48c541129ddd4c0a9fcbf5f3c6a18510" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:56.486564 18953 mvcc.cc:205] Tried to move back new op lower bound from 7333265311672029184 to 7333265311589748736. Current Snapshot: MvccSnapshot[applied={T|T < 7333265311672029184}]
I20260925 14:52:56.486927 18952 mvcc.cc:205] Tried to move back new op lower bound from 7333265311672029184 to 7333265311589748736. Current Snapshot: MvccSnapshot[applied={T|T < 7333265311672029184}]
I20260925 14:52:56.488448 18461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:56.488548 18461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:56.489305 18461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e17d5d909694e3cac86e86c6ec3d98c: Sending DeleteTablet for 3 replicas of tablet 2f446e7d2d94481a9492c38d7e65789f
I20260925 14:52:56.489636 18721 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f446e7d2d94481a9492c38d7e65789f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51718
I20260925 14:52:56.489643 18855 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f446e7d2d94481a9492c38d7e65789f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32804
I20260925 14:52:56.489861 18961 tablet_replica.cc:330] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: stopping tablet replica
I20260925 14:52:56.489964 18961 raft_consensus.cc:2229] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:56.489985 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f446e7d2d94481a9492c38d7e65789f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40616
I20260925 14:52:56.490064 18961 raft_consensus.cc:2258] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:56.490314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18524
I20260925 14:52:56.491259 18962 tablet_replica.cc:330] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: stopping tablet replica
I20260925 14:52:56.491343 18962 raft_consensus.cc:2229] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:56.491411 18962 raft_consensus.cc:2258] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:56.491786 18962 ts_tablet_manager.cc:1905] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:56.493250 18962 ts_tablet_manager.cc:1918] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:56.493300 18962 log.cc:1196] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/2f446e7d2d94481a9492c38d7e65789f
I20260925 14:52:56.493494 18962 ts_tablet_manager.cc:1939] T 2f446e7d2d94481a9492c38d7e65789f P 48f343a8ba144f2d926854fb5d26931d: Deleting consensus metadata
I20260925 14:52:56.493824 18448 catalog_manager.cc:4841] TS 48f343a8ba144f2d926854fb5d26931d (127.0.98.3:40703): tablet 2f446e7d2d94481a9492c38d7e65789f (table test-table [id=3ea7d787e6b64c12a681ca0525b1b272]) successfully deleted
I20260925 14:52:56.496457 18961 ts_tablet_manager.cc:1905] T 2f446e7d2d94481a9492c38d7e65789f P f53774033e9e4fd5abe508e0708b2b11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 14:52:56.497263 18449 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42975 (error 108)
I20260925 14:52:56.497442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18658
I20260925 14:52:56.499840 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.499604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42975 (local address 127.0.0.1:40630)
0925 14:52:56.499718 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.499805 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260925 14:52:56.500021 18449 catalog_manager.cc:4584] TS 48c541129ddd4c0a9fcbf5f3c6a18510 (127.0.98.1:42975): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2f446e7d2d94481a9492c38d7e65789f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42975: connect: Connection refused (error 111)
I20260925 14:52:56.503135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18792
I20260925 14:52:56.503202 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.503079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43203 (local address 127.0.0.1:51750)
0925 14:52:56.503146 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.503173 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 14:52:56.507462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18430
2026-09-25T14:52:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2172 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T14:52:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:56Z Disabled control of system clock
I20260925 14:52:56.536628 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36209
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:56.621165 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:56.621392 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:56.621414 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:56.622712 18968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:56.622756 18968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:56.622771 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:56.622782 18968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:56.622795 18968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:56.624094 18968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36209
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:56.624289 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:56.624517 18968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:56.629642 18974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.629650 18976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.629650 18973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:56.629801 18968 server_base.cc:1034] running on GCE node
I20260925 14:52:56.630263 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:56.630607 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:56.631779 18968 hybrid_clock.cc:648] HybridClock initialized: now 1790347976631768 us; error 32 us; skew 500 ppm
I20260925 14:52:56.633016 18968 webserver.cc:460] Webserver started at http://127.0.98.62:38645/ using document root <none> and password file <none>
I20260925 14:52:56.633267 18968 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:56.633337 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:56.633471 18968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:56.634630 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "35ba8576a0e54fd3a458c3bf82c8321e"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "b4c6f274b906e92d45002c6d155941fd"
server_key_iv: "3eb494ab40040c5ad40b140ee83513f2"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.634999 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "35ba8576a0e54fd3a458c3bf82c8321e"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "b4c6f274b906e92d45002c6d155941fd"
server_key_iv: "3eb494ab40040c5ad40b140ee83513f2"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.636292 18968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:56.637032 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.637198 18968 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:56.637298 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "35ba8576a0e54fd3a458c3bf82c8321e"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "b4c6f274b906e92d45002c6d155941fd"
server_key_iv: "3eb494ab40040c5ad40b140ee83513f2"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.637382 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:56.644771 18968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:56.645110 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:56.645275 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:56.649303 18968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36209
I20260925 14:52:56.649334 19034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36209 every 8 connection(s)
I20260925 14:52:56.649708 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:56.650780 19035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:56.650998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 18968
I20260925 14:52:56.651088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:56.651316 392 external_mini_cluster.cc:1428] Setting key 9eecd85e932cc3076f2a06473f736bd7
I20260925 14:52:56.653401 19035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e: Bootstrap starting.
I20260925 14:52:56.654042 19035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:56.654340 19035 log.cc:824] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:56.655128 19035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e: No bootstrap required, opened a new log
I20260925 14:52:56.657099 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.651888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.0.98.62:36209)
0925 14:52:56.652110 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.652118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.652142 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:52:56.652581 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.652588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.652648 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.652820 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.653049 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.654012 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.655089 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.655567 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.655721 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.656490 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.656494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.656494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:56.656656 (+ 162us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.656689 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:56.656701 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.656788 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.656863 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.656949 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260925 14:52:56.657176 19035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } }
I20260925 14:52:56.657368 19035 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:56.657403 19035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ba8576a0e54fd3a458c3bf82c8321e, State: Initialized, Role: FOLLOWER
I20260925 14:52:56.657754 19035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } }
I20260925 14:52:56.657851 19035 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:56.657903 19035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:56.657958 19035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:56.658581 19035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } }
I20260925 14:52:56.658702 19035 leader_election.cc:302] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35ba8576a0e54fd3a458c3bf82c8321e; no voters:
I20260925 14:52:56.658892 19035 leader_election.cc:288] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:56.658982 19040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:56.659134 19040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [term 1 LEADER]: Becoming Leader. State: Replica: 35ba8576a0e54fd3a458c3bf82c8321e, State: Running, Role: LEADER
I20260925 14:52:56.659162 19035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:56.659255 19040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } }
I20260925 14:52:56.659724 19041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } }
I20260925 14:52:56.659798 19041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:56.659700 19042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35ba8576a0e54fd3a458c3bf82c8321e. Latest consensus state: current_term: 1 leader_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35ba8576a0e54fd3a458c3bf82c8321e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36209 } } }
I20260925 14:52:56.659888 19042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:56.660257 19049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:56.660827 19049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:56.662371 19049 catalog_manager.cc:1347] Generated new cluster ID: cc087399a5a34fbea3b73e9d51de0d6c
I20260925 14:52:56.662467 19049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:56.684159 19049 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:56.684712 19049 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:56.690331 19049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e: Generated new TSK 0
I20260925 14:52:56.690553 19049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:56.697680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36209
--builtin_ntp_servers=127.0.98.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:56.780479 19059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:56.780694 19059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:56.780717 19059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:56.781919 19059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:56.781996 19059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:56.782042 19059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:56.783448 19059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:56.783649 19059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:56.783903 19059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:56.788831 19067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.788831 19064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.788867 19065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:56.789211 19059 server_base.cc:1034] running on GCE node
I20260925 14:52:56.789422 19059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:56.789615 19059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:56.790758 19059 hybrid_clock.cc:648] HybridClock initialized: now 1790347976790739 us; error 40 us; skew 500 ppm
I20260925 14:52:56.792078 19059 webserver.cc:460] Webserver started at http://127.0.98.1:38811/ using document root <none> and password file <none>
I20260925 14:52:56.792348 19059 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:56.792397 19059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:56.792488 19059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:56.793505 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9d305319a6d47b6a2c98064fadf720c"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "ba2f7b7c1a7d1e4f930b14567d67e8ad"
server_key_iv: "41bd8cc8580ffcf743383b8ac1094471"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.793804 19059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9d305319a6d47b6a2c98064fadf720c"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "ba2f7b7c1a7d1e4f930b14567d67e8ad"
server_key_iv: "41bd8cc8580ffcf743383b8ac1094471"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.795094 19059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:52:56.795768 19073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.795933 19059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:52:56.796001 19059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "d9d305319a6d47b6a2c98064fadf720c"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "ba2f7b7c1a7d1e4f930b14567d67e8ad"
server_key_iv: "41bd8cc8580ffcf743383b8ac1094471"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.796052 19059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:56.812726 19059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:56.812997 19059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:56.813093 19059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:56.813284 19059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:56.813611 19059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:56.813645 19059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.813700 19059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:56.813724 19059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.819480 19059 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42561
I20260925 14:52:56.819527 19186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42561 every 8 connection(s)
I20260925 14:52:56.819855 19059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:56.822745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19059
I20260925 14:52:56.822849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:56.823052 392 external_mini_cluster.cc:1428] Setting key 9005515630573465b9213e7c574dc287
I20260925 14:52:56.823978 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36209
--builtin_ntp_servers=127.0.98.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:52:56.825631 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.820564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55097 (local address 127.0.98.62:36209)
0925 14:52:56.820627 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.820631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.820866 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:52:56.821342 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.821343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.821356 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.821406 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.822066 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.822596 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.824078 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.824330 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.824385 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.825303 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.825305 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.825306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:56.825311 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.825329 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:56.825334 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.825396 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.825493 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.825527 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:52:56.825755 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.820552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36209 (local address 127.0.98.1:55097)
0925 14:52:56.820747 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.820781 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.821262 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.821465 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.821469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.821476 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:56.821961 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.821965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.822688 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.822690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.823978 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.823981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.824416 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.824417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.824581 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.825126 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:56.825127 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:56.825169 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:56.825200 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:56.825404 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:56.825409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:56.825413 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.825476 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.825554 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260925 14:52:56.826210 19187 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36209
I20260925 14:52:56.826314 19187 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.826534 19187 heartbeater.cc:511] Master 127.0.98.62:36209 requested a full tablet report, sending...
I20260925 14:52:56.827032 18999 ts_manager.cc:194] Registered new tserver with Master: d9d305319a6d47b6a2c98064fadf720c (127.0.98.1:42561)
I20260925 14:52:56.827718 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55097
W20260925 14:52:56.905385 19190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:56.905593 19190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:56.905616 19190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:56.906733 19190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:56.906767 19190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:56.906791 19190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:56.908103 19190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:56.908340 19190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:56.908601 19190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:56.913751 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.913715 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:56.913715 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:56.913757 19190 server_base.cc:1034] running on GCE node
I20260925 14:52:56.914171 19190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:56.914474 19190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:56.915649 19190 hybrid_clock.cc:648] HybridClock initialized: now 1790347976915635 us; error 31 us; skew 500 ppm
I20260925 14:52:56.916975 19190 webserver.cc:460] Webserver started at http://127.0.98.2:41425/ using document root <none> and password file <none>
I20260925 14:52:56.917235 19190 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:56.917310 19190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:56.917464 19190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:56.918614 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "129339b0148444898798816be56d84f5"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "b1f638daef338e8b53c408433c2a520e"
server_key_iv: "414892153a52dba1e22d03b318682961"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.918995 19190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "129339b0148444898798816be56d84f5"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "b1f638daef338e8b53c408433c2a520e"
server_key_iv: "414892153a52dba1e22d03b318682961"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.920282 19190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:56.921010 19204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.921187 19190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:56.921293 19190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "129339b0148444898798816be56d84f5"
format_stamp: "Formatted at 2026-09-25 14:52:56 on dist-test-slave-z2s5"
server_key: "b1f638daef338e8b53c408433c2a520e"
server_key_iv: "414892153a52dba1e22d03b318682961"
server_key_version: "encryptionkey@0"
I20260925 14:52:56.921368 19190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:56.940653 19190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:56.940996 19190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:56.941150 19190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:56.941382 19190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:56.941736 19190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:56.941793 19190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.941828 19190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:56.941854 19190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:56.947429 19190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44339
I20260925 14:52:56.947470 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44339 every 8 connection(s)
I20260925 14:52:56.947826 19190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:56.949363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19190
I20260925 14:52:56.949465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:56.949664 392 external_mini_cluster.cc:1428] Setting key 9bdc12f0c519a4a179ee226916007824
I20260925 14:52:56.950721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36209
--builtin_ntp_servers=127.0.98.20:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:52:56.953557 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.948612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49305 (local address 127.0.98.62:36209)
0925 14:52:56.948680 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:56.948682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:56.948914 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:52:56.949402 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:56.949404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:56.949418 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:56.949504 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:56.950341 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.951110 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.952089 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.952300 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.952388 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.953269 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:56.953270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:56.953271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:56.953276 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:56.953291 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:56.953295 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:56.953350 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:56.953450 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:52:56.953478 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:52:56.953796 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:56.948601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36209 (local address 127.0.98.2:49305)
0925 14:52:56.948780 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:56.948822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:56.949303 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:56.949590 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:56.949601 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:56.949610 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:56.950243 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.950249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.951201 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.951202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.951988 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:56.951990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:56.952400 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:56.952401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:56.952538 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:56.953119 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:56.953119 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:56.953149 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:56.953171 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:56.953353 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:56.953358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:56.953361 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:56.953453 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 14:52:56.953556 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260925 14:52:56.954334 19318 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36209
I20260925 14:52:56.954465 19318 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:56.954684 19318 heartbeater.cc:511] Master 127.0.98.62:36209 requested a full tablet report, sending...
I20260925 14:52:56.955091 18999 ts_manager.cc:194] Registered new tserver with Master: 129339b0148444898798816be56d84f5 (127.0.98.2:44339)
I20260925 14:52:56.955646 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49305
W20260925 14:52:57.033385 19321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:57.033661 19321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:57.033686 19321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:57.034888 19321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:57.034946 19321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:57.034991 19321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:57.036298 19321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:57.036480 19321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:57.036732 19321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:57.041698 19326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:57.041759 19329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:57.041909 19327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:57.041918 19321 server_base.cc:1034] running on GCE node
I20260925 14:52:57.042196 19321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:57.042497 19321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:57.043692 19321 hybrid_clock.cc:648] HybridClock initialized: now 1790347977043676 us; error 48 us; skew 500 ppm
I20260925 14:52:57.045037 19321 webserver.cc:460] Webserver started at http://127.0.98.3:34827/ using document root <none> and password file <none>
I20260925 14:52:57.045324 19321 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:57.045423 19321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:57.045523 19321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:57.046631 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "518f7b6d318b48128803bc0d466dff5b"
format_stamp: "Formatted at 2026-09-25 14:52:57 on dist-test-slave-z2s5"
server_key: "1d2b83be01aacb2c0bc1c2c719d44b67"
server_key_iv: "ab8278334365cd701e8d9d1a6c16cfdc"
server_key_version: "encryptionkey@0"
I20260925 14:52:57.046974 19321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "518f7b6d318b48128803bc0d466dff5b"
format_stamp: "Formatted at 2026-09-25 14:52:57 on dist-test-slave-z2s5"
server_key: "1d2b83be01aacb2c0bc1c2c719d44b67"
server_key_iv: "ab8278334365cd701e8d9d1a6c16cfdc"
server_key_version: "encryptionkey@0"
I20260925 14:52:57.048174 19321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:57.048910 19335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:57.049146 19321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:57.049213 19321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "518f7b6d318b48128803bc0d466dff5b"
format_stamp: "Formatted at 2026-09-25 14:52:57 on dist-test-slave-z2s5"
server_key: "1d2b83be01aacb2c0bc1c2c719d44b67"
server_key_iv: "ab8278334365cd701e8d9d1a6c16cfdc"
server_key_version: "encryptionkey@0"
I20260925 14:52:57.049310 19321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:57.067474 19321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:57.067824 19321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:57.067986 19321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:57.068252 19321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:57.068637 19321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:57.068694 19321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:57.068738 19321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:57.068766 19321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:57.074561 19321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39537
I20260925 14:52:57.074601 19448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39537 every 8 connection(s)
I20260925 14:52:57.074975 19321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:57.075665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19321
I20260925 14:52:57.075766 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:57.075959 392 external_mini_cluster.cc:1428] Setting key 3701a9942b80e10621ebe8ed33fe614d
I20260925 14:52:57.080104 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.075759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54485 (local address 127.0.98.62:36209)
0925 14:52:57.075823 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.075825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.076055 (+ 230us) server_negotiation.cc:407] Connection header received
0925 14:52:57.076662 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.076663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.076672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.076709 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:57.077385 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.077869 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.078889 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.079068 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.079120 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.079881 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:57.079882 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:57.079882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:57.079886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:57.079900 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:57.079903 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:57.079946 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.080026 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.080046 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:52:57.080296 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.075705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36209 (local address 127.0.98.3:54485)
0925 14:52:57.075912 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.075954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.076559 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.076805 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.076809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.076817 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:57.077311 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.077314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.077951 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.077952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.078819 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.078821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.079139 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.079140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.079248 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.079756 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:57.079756 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:57.079785 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:57.079807 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:57.079958 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:57.079962 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:57.079964 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.080016 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.080092 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260925 14:52:57.080746 19449 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36209
I20260925 14:52:57.080822 19449 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:57.080958 19449 heartbeater.cc:511] Master 127.0.98.62:36209 requested a full tablet report, sending...
I20260925 14:52:57.081286 18999 ts_manager.cc:194] Registered new tserver with Master: 518f7b6d318b48128803bc0d466dff5b (127.0.98.3:39537)
I20260925 14:52:57.081609 18999 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:54485
I20260925 14:52:57.087790 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:57.091935 19038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.089265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.0.98.62:36209)
0925 14:52:57.089329 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.089331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.089426 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:52:57.089463 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.089464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.089473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.089521 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:57.089656 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.090113 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.090854 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.091030 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.091081 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.091697 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:57.091698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:57.091698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:57.091702 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:57.091714 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:57.091718 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:57.091798 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.091869 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.091880 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:52:57.093439 18999 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:57.095129 18999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:57.101622 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.097400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39537 (local address 127.0.0.1:41952)
0925 14:52:57.097712 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.097724 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.097783 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.098001 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.098004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.098006 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.098083 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.098085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.099228 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.099229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.100467 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.100469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.101309 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.101310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.101385 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.101399 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.101482 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.101545 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":99,"threads_started":1}
I20260925 14:52:57.101632 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.097515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.0.98.3:39537)
0925 14:52:57.097786 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.097791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.097814 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:52:57.097859 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.097861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.097880 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.098169 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.098235 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.099124 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.100581 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.101193 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.101304 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.101399 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.101497 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.101539 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":98,"threads_started":1}
I20260925 14:52:57.101883 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.097810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.0.98.1:42561)
0925 14:52:57.097991 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.097997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.098005 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:57.098041 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.098043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.098056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.098128 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.098310 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.099535 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.100612 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.101358 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.101429 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.101519 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.101747 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.101787 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260925 14:52:57.101799 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.097359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44339 (local address 127.0.0.1:49962)
0925 14:52:57.097562 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.097602 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.097667 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.098127 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.098128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.098130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.098175 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.098177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.099614 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.099615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.100700 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.100702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.101573 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.101574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.101625 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.101628 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.101658 (+ 30us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.101702 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":28,"thread_start_us":94,"threads_started":1}
I20260925 14:52:57.102042 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.097602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42561 (local address 127.0.0.1:47274)
0925 14:52:57.097872 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.097881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.097922 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.098194 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.098195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.098197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.098232 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.098233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.099622 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.099623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.100525 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.100527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.101555 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.101556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.101613 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.101617 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.101667 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.101702 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260925 14:52:57.102324 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.097578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49962 (local address 127.0.98.2:44339)
0925 14:52:57.097859 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.097864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.097876 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:52:57.097950 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.097953 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.097969 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.098265 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.098330 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.099516 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.100791 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.101483 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.101872 (+ 389us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.101973 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.102001 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.102040 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":75,"threads_started":1}
I20260925 14:52:57.102829 19383 tablet_service.cc:1467] Processing CreateTablet for tablet bdbd637dfb3f434dab221d6f174bd80f (DEFAULT_TABLE table=test-table [id=31fb8f2938014ec795a87b1b9192d6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:57.102979 19121 tablet_service.cc:1467] Processing CreateTablet for tablet bdbd637dfb3f434dab221d6f174bd80f (DEFAULT_TABLE table=test-table [id=31fb8f2938014ec795a87b1b9192d6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:57.103163 19383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbd637dfb3f434dab221d6f174bd80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:57.103350 19121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbd637dfb3f434dab221d6f174bd80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:57.105756 19463 tablet_bootstrap.cc:492] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Bootstrap starting.
I20260925 14:52:57.106237 19252 tablet_service.cc:1467] Processing CreateTablet for tablet bdbd637dfb3f434dab221d6f174bd80f (DEFAULT_TABLE table=test-table [id=31fb8f2938014ec795a87b1b9192d6f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:57.106545 19463 tablet_bootstrap.cc:654] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:57.106619 19252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdbd637dfb3f434dab221d6f174bd80f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:57.106637 19464 tablet_bootstrap.cc:492] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: Bootstrap starting.
I20260925 14:52:57.106889 19463 log.cc:824] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:57.107286 19464 tablet_bootstrap.cc:654] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:57.107590 19464 log.cc:824] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:57.107748 19463 tablet_bootstrap.cc:492] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: No bootstrap required, opened a new log
I20260925 14:52:57.107831 19463 ts_tablet_manager.cc:1397] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:52:57.110173 19467 tablet_bootstrap.cc:492] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Bootstrap starting.
I20260925 14:52:57.110811 19467 tablet_bootstrap.cc:654] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:57.110790 19463 raft_consensus.cc:348] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.110993 19463 raft_consensus.cc:374] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:57.111050 19463 raft_consensus.cc:729] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518f7b6d318b48128803bc0d466dff5b, State: Initialized, Role: FOLLOWER
I20260925 14:52:57.111133 19464 tablet_bootstrap.cc:492] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: No bootstrap required, opened a new log
I20260925 14:52:57.111187 19467 log.cc:824] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:57.111207 19464 ts_tablet_manager.cc:1397] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:52:57.111178 19463 consensus_queue.cc:260] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.111445 19463 ts_tablet_manager.cc:1428] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260925 14:52:57.111558 19449 heartbeater.cc:503] Master 127.0.98.62:36209 was elected leader, sending a full tablet report...
I20260925 14:52:57.112098 19467 tablet_bootstrap.cc:492] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: No bootstrap required, opened a new log
I20260925 14:52:57.112170 19467 ts_tablet_manager.cc:1397] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:57.113658 19464 raft_consensus.cc:348] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.113842 19464 raft_consensus.cc:374] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:57.113898 19464 raft_consensus.cc:729] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d305319a6d47b6a2c98064fadf720c, State: Initialized, Role: FOLLOWER
I20260925 14:52:57.114044 19464 consensus_queue.cc:260] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.114095 19467 raft_consensus.cc:348] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.114264 19467 raft_consensus.cc:374] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:57.114315 19467 raft_consensus.cc:729] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 129339b0148444898798816be56d84f5, State: Initialized, Role: FOLLOWER
I20260925 14:52:57.114375 19464 ts_tablet_manager.cc:1428] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 14:52:57.114420 19187 heartbeater.cc:503] Master 127.0.98.62:36209 was elected leader, sending a full tablet report...
I20260925 14:52:57.114472 19467 consensus_queue.cc:260] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.114753 19467 ts_tablet_manager.cc:1428] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:52:57.114974 19318 heartbeater.cc:503] Master 127.0.98.62:36209 was elected leader, sending a full tablet report...
W20260925 14:52:57.198465 19319 tablet.cc:2369] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:57.320513 19188 tablet.cc:2369] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:57.325538 19450 tablet.cc:2369] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:57.398483 19471 raft_consensus.cc:482] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:57.398645 19471 raft_consensus.cc:504] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.399070 19471 leader_election.cc:288] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 518f7b6d318b48128803bc0d466dff5b (127.0.98.3:39537), d9d305319a6d47b6a2c98064fadf720c (127.0.98.1:42561)
I20260925 14:52:57.402051 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.399350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53589 (local address 127.0.98.3:39537)
0925 14:52:57.399400 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.399402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.399416 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:52:57.399448 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.399449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.399459 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.399547 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.399760 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.400327 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.401276 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.401693 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.401782 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.401793 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.401940 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.401981 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:52:57.402051 19320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.399262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39537 (local address 127.0.98.2:53589)
0925 14:52:57.399304 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.399317 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.399374 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.399554 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.399555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.399557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.399672 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.399674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.400421 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.400422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.401175 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.401176 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.401781 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.401782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.401860 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.401862 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.401933 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.401978 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 14:52:57.402129 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.399403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49205 (local address 127.0.98.1:42561)
0925 14:52:57.399448 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.399450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.399555 (+ 105us) server_negotiation.cc:407] Connection header received
0925 14:52:57.399620 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.399621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.399631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.399709 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:57.399921 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.400374 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.401340 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.401735 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.401830 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.401840 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.402015 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.402053 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:52:57.402412 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.399304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42561 (local address 127.0.98.2:49205)
0925 14:52:57.399494 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.399502 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:57.399565 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:57.399714 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:57.399714 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:57.399716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:57.399795 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.399812 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.400463 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.400464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.401210 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:57.401212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.401834 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.401835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:57.401925 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.401928 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:57.402005 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:52:57.402043 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":104,"threads_started":1}
I20260925 14:52:57.402468 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbd637dfb3f434dab221d6f174bd80f" candidate_uuid: "129339b0148444898798816be56d84f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518f7b6d318b48128803bc0d466dff5b" is_pre_election: true
I20260925 14:52:57.402654 19403 raft_consensus.cc:2454] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129339b0148444898798816be56d84f5 in term 0.
I20260925 14:52:57.402784 19141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbd637dfb3f434dab221d6f174bd80f" candidate_uuid: "129339b0148444898798816be56d84f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d305319a6d47b6a2c98064fadf720c" is_pre_election: true
I20260925 14:52:57.402971 19141 raft_consensus.cc:2454] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 129339b0148444898798816be56d84f5 in term 0.
I20260925 14:52:57.402959 19208 leader_election.cc:302] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129339b0148444898798816be56d84f5, 518f7b6d318b48128803bc0d466dff5b; no voters:
I20260925 14:52:57.403124 19471 raft_consensus.cc:2781] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:57.403178 19471 raft_consensus.cc:482] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:57.403196 19471 raft_consensus.cc:3037] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.403941 19471 raft_consensus.cc:504] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.404089 19471 leader_election.cc:288] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [CANDIDATE]: Term 1 election: Requested vote from peers 518f7b6d318b48128803bc0d466dff5b (127.0.98.3:39537), d9d305319a6d47b6a2c98064fadf720c (127.0.98.1:42561)
I20260925 14:52:57.404260 19403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbd637dfb3f434dab221d6f174bd80f" candidate_uuid: "129339b0148444898798816be56d84f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518f7b6d318b48128803bc0d466dff5b"
I20260925 14:52:57.404268 19141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdbd637dfb3f434dab221d6f174bd80f" candidate_uuid: "129339b0148444898798816be56d84f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d305319a6d47b6a2c98064fadf720c"
I20260925 14:52:57.404341 19403 raft_consensus.cc:3037] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.404366 19141 raft_consensus.cc:3037] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:57.405058 19403 raft_consensus.cc:2454] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129339b0148444898798816be56d84f5 in term 1.
I20260925 14:52:57.405181 19141 raft_consensus.cc:2454] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 129339b0148444898798816be56d84f5 in term 1.
I20260925 14:52:57.405340 19208 leader_election.cc:302] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 129339b0148444898798816be56d84f5, 518f7b6d318b48128803bc0d466dff5b; no voters:
I20260925 14:52:57.405429 19471 raft_consensus.cc:2781] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:57.405587 19471 raft_consensus.cc:686] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 1 LEADER]: Becoming Leader. State: Replica: 129339b0148444898798816be56d84f5, State: Running, Role: LEADER
I20260925 14:52:57.405666 19471 consensus_queue.cc:237] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } }
I20260925 14:52:57.406545 18998 catalog_manager.cc:5495] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 129339b0148444898798816be56d84f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "129339b0148444898798816be56d84f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "129339b0148444898798816be56d84f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:57.458511 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.456137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.0.98.2:44339)
0925 14:52:57.456207 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:57.456209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:57.456219 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:52:57.456277 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:57.456277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:57.456298 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:57.456381 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:57.456518 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.456980 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.457722 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:57.457919 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:57.457996 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:57.458147 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:57.458234 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:57.458351 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:57.458402 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:52:57.458412 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:52:57.460474 19141 raft_consensus.cc:1261] T bdbd637dfb3f434dab221d6f174bd80f P d9d305319a6d47b6a2c98064fadf720c [term 1 FOLLOWER]: Refusing update from remote peer 129339b0148444898798816be56d84f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:57.460479 19403 raft_consensus.cc:1261] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 1 FOLLOWER]: Refusing update from remote peer 129339b0148444898798816be56d84f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:57.460865 19473 consensus_queue.cc:1035] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d305319a6d47b6a2c98064fadf720c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:57.461037 19471 consensus_queue.cc:1035] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "518f7b6d318b48128803bc0d466dff5b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:57.463760 19476 mvcc.cc:205] Tried to move back new op lower bound from 7333265315675009024 to 7333265315454218240. Current Snapshot: MvccSnapshot[applied={T|T < 7333265315675009024}]
I20260925 14:52:57.463979 19477 mvcc.cc:205] Tried to move back new op lower bound from 7333265315675009024 to 7333265315454218240. Current Snapshot: MvccSnapshot[applied={T|T < 7333265315675009024}]
I20260925 14:52:57.466120 18997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:57.466228 18997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:57.467053 18997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35ba8576a0e54fd3a458c3bf82c8321e: Sending DeleteTablet for 3 replicas of tablet bdbd637dfb3f434dab221d6f174bd80f
I20260925 14:52:57.467335 19252 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbd637dfb3f434dab221d6f174bd80f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='slave'} at 127.0.0.1:49962
I20260925 14:52:57.467381 19121 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbd637dfb3f434dab221d6f174bd80f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='slave'} at 127.0.0.1:47274
I20260925 14:52:57.467386 19383 tablet_service.cc:1514] Processing DeleteTablet for tablet bdbd637dfb3f434dab221d6f174bd80f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:57 UTC) from {username='slave'} at 127.0.0.1:41952
I20260925 14:52:57.467592 19488 tablet_replica.cc:330] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: stopping tablet replica
I20260925 14:52:57.467682 19486 tablet_replica.cc:330] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: stopping tablet replica
I20260925 14:52:57.467692 19488 raft_consensus.cc:2229] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:57.467758 19486 raft_consensus.cc:2229] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:57.467809 19488 raft_consensus.cc:2258] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.467964 19486 raft_consensus.cc:2258] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:57.468065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19059
I20260925 14:52:57.469065 19486 ts_tablet_manager.cc:1905] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.471194 19486 ts_tablet_manager.cc:1918] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:57.471350 19486 log.cc:1196] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/bdbd637dfb3f434dab221d6f174bd80f
I20260925 14:52:57.471678 19486 ts_tablet_manager.cc:1939] T bdbd637dfb3f434dab221d6f174bd80f P 129339b0148444898798816be56d84f5: Deleting consensus metadata
I20260925 14:52:57.472128 18985 catalog_manager.cc:4841] TS 129339b0148444898798816be56d84f5 (127.0.98.2:44339): tablet bdbd637dfb3f434dab221d6f174bd80f (table test-table [id=31fb8f2938014ec795a87b1b9192d6f3]) successfully deleted
I20260925 14:52:57.472350 19488 ts_tablet_manager.cc:1905] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:57.474021 19488 ts_tablet_manager.cc:1918] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:57.474107 19488 log.cc:1196] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/bdbd637dfb3f434dab221d6f174bd80f
I20260925 14:52:57.474433 19488 ts_tablet_manager.cc:1939] T bdbd637dfb3f434dab221d6f174bd80f P 518f7b6d318b48128803bc0d466dff5b: Deleting consensus metadata
I20260925 14:52:57.474852 18986 catalog_manager.cc:4841] TS 518f7b6d318b48128803bc0d466dff5b (127.0.98.3:39537): tablet bdbd637dfb3f434dab221d6f174bd80f (table test-table [id=31fb8f2938014ec795a87b1b9192d6f3]) successfully deleted
W20260925 14:52:57.475319 18985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42561 (error 108)
I20260925 14:52:57.475502 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19190
I20260925 14:52:57.476616 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:57.476490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42561 (local address 127.0.0.1:47290)
0925 14:52:57.476539 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:57.476576 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42561: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 14:52:57.476833 18985 catalog_manager.cc:4584] TS d9d305319a6d47b6a2c98064fadf720c (127.0.98.1:42561): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdbd637dfb3f434dab221d6f174bd80f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42561: connect: Connection refused (error 111)
I20260925 14:52:57.480183 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19321
I20260925 14:52:57.484702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 18968
2026-09-25T14:52:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (975 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:52:57 dist-test-slave-z2s5 krb5kdc[19492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:52:57 dist-test-slave-z2s5 krb5kdc[19492](info): set up 2 sockets
Sep 25 14:52:57 dist-test-slave-z2s5 krb5kdc[19492](info): commencing operation
krb5kdc: starting...
W20260925 14:52:58.538261 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:52:58 dist-test-slave-z2s5 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:52:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:52:58.657187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:58.737604 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:58.737852 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:58.737892 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:58.739070 19505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:58.739144 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:58.739168 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:58.739197 19505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:58.739228 19505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:58.740723 19505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:58.741005 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:58.741242 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:58.746152 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:58.746343 19505 server_base.cc:1034] running on GCE node
W20260925 14:52:58.746152 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:58.746194 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:58.746820 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:58.747129 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:58.748337 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790347978748325 us; error 51 us; skew 500 ppm
Sep 25 14:52:58 dist-test-slave-z2s5 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:58.751194 19505 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:52:58.751725 19505 webserver.cc:460] Webserver started at http://127.0.98.62:46471/ using document root <none> and password file <none>
I20260925 14:52:58.751986 19505 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:58.752058 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:58.752215 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:58.753334 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "1557c45dc193489f988b44dbb25c001c"
format_stamp: "Formatted at 2026-09-25 14:52:58 on dist-test-slave-z2s5"
server_key: "b8746f93c85e332036281959886e7a17"
server_key_iv: "f28eb317c321c10afd6549e0a1c09715"
server_key_version: "encryptionkey@0"
I20260925 14:52:58.753785 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "1557c45dc193489f988b44dbb25c001c"
format_stamp: "Formatted at 2026-09-25 14:52:58 on dist-test-slave-z2s5"
server_key: "b8746f93c85e332036281959886e7a17"
server_key_iv: "f28eb317c321c10afd6549e0a1c09715"
server_key_version: "encryptionkey@0"
I20260925 14:52:58.755290 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260925 14:52:58.756098 19520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:58.756389 19505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:58.756507 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "1557c45dc193489f988b44dbb25c001c"
format_stamp: "Formatted at 2026-09-25 14:52:58 on dist-test-slave-z2s5"
server_key: "b8746f93c85e332036281959886e7a17"
server_key_iv: "f28eb317c321c10afd6549e0a1c09715"
server_key_version: "encryptionkey@0"
I20260925 14:52:58.756599 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:58.777246 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:58.779692 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:58.779860 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:58.783857 19505 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46795
I20260925 14:52:58.783913 19572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46795 every 8 connection(s)
I20260925 14:52:58.784279 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:58.785195 19573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:58.787586 19573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c: Bootstrap starting.
I20260925 14:52:58.788170 19573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:58.788484 19573 log.cc:824] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:58.789224 19573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c: No bootstrap required, opened a new log
I20260925 14:52:58.790849 19573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1557c45dc193489f988b44dbb25c001c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260925 14:52:58.790987 19573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:58.791010 19573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1557c45dc193489f988b44dbb25c001c, State: Initialized, Role: FOLLOWER
I20260925 14:52:58.791193 19573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1557c45dc193489f988b44dbb25c001c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260925 14:52:58.791258 19573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:58.791280 19573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:58.791352 19573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:58.791868 19573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1557c45dc193489f988b44dbb25c001c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260925 14:52:58.791980 19573 leader_election.cc:302] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1557c45dc193489f988b44dbb25c001c; no voters:
I20260925 14:52:58.792191 19573 leader_election.cc:288] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:58.792217 19576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:58.792230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19505
I20260925 14:52:58.792337 19576 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [term 1 LEADER]: Becoming Leader. State: Replica: 1557c45dc193489f988b44dbb25c001c, State: Running, Role: LEADER
I20260925 14:52:58.792373 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:58.792454 19573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:58.792519 19576 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1557c45dc193489f988b44dbb25c001c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } }
I20260925 14:52:58.792572 392 external_mini_cluster.cc:1428] Setting key 925e45b9e274190a1c023373a244503d
I20260925 14:52:58.793090 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1557c45dc193489f988b44dbb25c001c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1557c45dc193489f988b44dbb25c001c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } } }
I20260925 14:52:58.793190 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:58.793417 19577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1557c45dc193489f988b44dbb25c001c. Latest consensus state: current_term: 1 leader_uuid: "1557c45dc193489f988b44dbb25c001c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1557c45dc193489f988b44dbb25c001c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46795 } } }
I20260925 14:52:58.793788 19577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:58.793985 19587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:58.794819 19587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:58.796437 19587 catalog_manager.cc:1347] Generated new cluster ID: ddded954405a4e01b455dd8adfcc934d
I20260925 14:52:58.796489 19587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:52:58 dist-test-slave-z2s5 krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:58.805301 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:58.793202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50834 (local address 127.0.98.62:46795)
0925 14:52:58.794086 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:58.794098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:58.794135 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:52:58.794260 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:58.794267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:58.794323 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:58.794520 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:58.795334 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:58.796714 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:58.797801 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:58.798489 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:58.798607 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:58.801441 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:58.801446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:58.801446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:58.801459 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:58.803557 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:58.803990 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:58.803993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:58.803993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:58.804121 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:58.804344 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:58.804345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:58.804345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:58.804658 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:58.804807 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:58.804932 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 14:52:58.805066 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":101,"threads_started":1}
I20260925 14:52:58.808393 19587 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:58.808945 19587 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:58.813474 19587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c: Generated new TSK 0
I20260925 14:52:58.813675 19587 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260925 14:52:58.843844 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:58.927627 19599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:58.927845 19599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:58.927868 19599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:58.929139 19599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:58.929188 19599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:58.929214 19599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:58.930764 19599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:58.931006 19599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:58.931263 19599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:58.936549 19607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:58.936620 19604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:58.936746 19599 server_base.cc:1034] running on GCE node
W20260925 14:52:58.936582 19605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:58.937016 19599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:58.937302 19599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:58.938479 19599 hybrid_clock.cc:648] HybridClock initialized: now 1790347978938470 us; error 41 us; skew 500 ppm
Sep 25 14:52:58 dist-test-slave-z2s5 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:58.940883 19599 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260925 14:52:58.941334 19599 webserver.cc:460] Webserver started at http://127.0.98.1:39809/ using document root <none> and password file <none>
I20260925 14:52:58.941574 19599 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:58.941643 19599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:58.941813 19599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:58.942936 19599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d29297f7f0141bfb8a89b19008f5d77"
format_stamp: "Formatted at 2026-09-25 14:52:58 on dist-test-slave-z2s5"
server_key: "15a2ed858cbe39f1e9a60ec022c6ec90"
server_key_iv: "e291600412b7fdfd96a8e038777a1ae3"
server_key_version: "encryptionkey@0"
I20260925 14:52:58.943295 19599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d29297f7f0141bfb8a89b19008f5d77"
format_stamp: "Formatted at 2026-09-25 14:52:58 on dist-test-slave-z2s5"
server_key: "15a2ed858cbe39f1e9a60ec022c6ec90"
server_key_iv: "e291600412b7fdfd96a8e038777a1ae3"
server_key_version: "encryptionkey@0"
I20260925 14:52:58.944608 19599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:58.945324 19614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:58.945497 19599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:58.945602 19599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "4d29297f7f0141bfb8a89b19008f5d77"
format_stamp: "Formatted at 2026-09-25 14:52:58 on dist-test-slave-z2s5"
server_key: "15a2ed858cbe39f1e9a60ec022c6ec90"
server_key_iv: "e291600412b7fdfd96a8e038777a1ae3"
server_key_version: "encryptionkey@0"
I20260925 14:52:58.945675 19599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:58.957211 19599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:58.959496 19599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:58.959664 19599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:58.959911 19599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:58.960246 19599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:58.960304 19599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:58.960345 19599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:58.960377 19599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:58.966367 19599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35125
I20260925 14:52:58.966487 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35125 every 8 connection(s)
I20260925 14:52:58.966814 19599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:58.969086 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19599
I20260925 14:52:58.969228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:58.969439 392 external_mini_cluster.cc:1428] Setting key 3f88c7afa69413dbc38c24ea08ecc6ba
Sep 25 14:52:58 dist-test-slave-z2s5 krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:58.978219 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:58.967719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46407 (local address 127.0.98.62:46795)
0925 14:52:58.967796 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:58.967798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:58.968037 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:52:58.969106 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:58.969107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:58.969125 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:58.969190 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:58.969986 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:58.970644 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:58.972117 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:58.972470 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:58.972556 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:58.975093 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:58.975095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:58.975095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:58.975105 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:58.976758 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:58.977165 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:58.977166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:58.977167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:58.977229 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:58.977495 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:58.977496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:58.977496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:58.977638 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:58.977720 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:58.977978 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 14:52:58.978079 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:52:58.978232 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:58.967696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.1:46407)
0925 14:52:58.967903 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:58.967941 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:58.968957 (+ 1016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:58.969272 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:58.969277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:58.969640 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:58.969890 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:58.969897 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:58.970745 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:58.970747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:58.972023 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:58.972026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:58.972568 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:58.972569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:58.972717 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:58.973293 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:58.973302 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:58.974984 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:58.976850 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:58.976856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:58.976858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:58.977077 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:58.977307 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:58.977308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:58.977308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:58.977414 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:58.977714 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:58.977720 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:58.977839 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:52:58.977926 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:52:58.978047 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260925 14:52:58.978895 19728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260925 14:52:58.979017 19728 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:58.979224 19728 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260925 14:52:58.979826 19537 ts_manager.cc:194] Registered new tserver with Master: 4d29297f7f0141bfb8a89b19008f5d77 (127.0.98.1:35125)
I20260925 14:52:58.980470 19537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260925 14:52:58.997356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:59.078622 19733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.078863 19733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.078900 19733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.080101 19733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.080179 19733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.080225 19733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:59.081753 19733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:59.082029 19733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.082273 19733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.087312 19738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.087285 19741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.087306 19739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.087414 19733 server_base.cc:1034] running on GCE node
I20260925 14:52:59.087808 19733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.088110 19733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.089300 19733 hybrid_clock.cc:648] HybridClock initialized: now 1790347979089278 us; error 44 us; skew 500 ppm
Sep 25 14:52:59 dist-test-slave-z2s5 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:59.092059 19733 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260925 14:52:59.092638 19733 webserver.cc:460] Webserver started at http://127.0.98.2:39973/ using document root <none> and password file <none>
I20260925 14:52:59.092849 19733 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.092895 19733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.093029 19733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.094226 19733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "64b87620ac134b70bcf6443290ec1b98"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "ff0544d5cab8e8b8383a24d141ca9abd"
server_key_iv: "f23872be17ad1611f36bdd82662cfeb8"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.094671 19733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "64b87620ac134b70bcf6443290ec1b98"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "ff0544d5cab8e8b8383a24d141ca9abd"
server_key_iv: "f23872be17ad1611f36bdd82662cfeb8"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.096055 19733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:59.096772 19748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.096958 19733 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:59.097060 19733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "64b87620ac134b70bcf6443290ec1b98"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "ff0544d5cab8e8b8383a24d141ca9abd"
server_key_iv: "f23872be17ad1611f36bdd82662cfeb8"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.097141 19733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.120391 19733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.122324 19733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.122520 19733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.122780 19733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:59.123123 19733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:59.123158 19733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.123180 19733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:59.123231 19733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.129200 19733 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34283
I20260925 14:52:59.129254 19861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34283 every 8 connection(s)
I20260925 14:52:59.129575 19733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:59.132496 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19733
I20260925 14:52:59.132633 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:59.132835 392 external_mini_cluster.cc:1428] Setting key d52f6effe092c29212100efb6be0b097
Sep 25 14:52:59 dist-test-slave-z2s5 krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:59.141315 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.130370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51887 (local address 127.0.98.62:46795)
0925 14:52:59.130486 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.130488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.130755 (+ 267us) server_negotiation.cc:407] Connection header received
0925 14:52:59.131680 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.131682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.131692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.131753 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.132517 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.133321 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.134964 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.135292 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.135373 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.137941 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.137943 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.137943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.137952 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.139807 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.140233 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.140235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.140235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.140311 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.140613 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.140614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.140614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.140763 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.140840 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.141089 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.141186 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 14:52:59.141330 19864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.130264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.2:51887)
0925 14:52:59.130601 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.130655 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.131577 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.131848 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.131852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.132120 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.132385 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.132393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.133404 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.133405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.134856 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.134859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.135384 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.135385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.135487 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.136052 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:59.136060 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.137825 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.139907 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.139913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.139915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.140147 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.140404 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.140405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.140405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.140531 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.140834 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.140839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.140957 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.141041 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.141168 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260925 14:52:59.141947 19862 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260925 14:52:59.142050 19862 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260925 14:52:59.142277 19862 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260925 14:52:59.142735 19537 ts_manager.cc:194] Registered new tserver with Master: 64b87620ac134b70bcf6443290ec1b98 (127.0.98.2:34283)
I20260925 14:52:59.143121 19537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260925 14:52:59.161510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:52:59.243114 19867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.243321 19867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.243345 19867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.244531 19867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.244575 19867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.244602 19867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:52:59.246203 19867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:59.246492 19867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.246737 19867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.251984 19873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.252002 19872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.251989 19875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.252374 19867 server_base.cc:1034] running on GCE node
I20260925 14:52:59.252584 19867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.252832 19867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.253999 19867 hybrid_clock.cc:648] HybridClock initialized: now 1790347979253984 us; error 36 us; skew 500 ppm
Sep 25 14:52:59 dist-test-slave-z2s5 krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:52:59.256636 19867 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260925 14:52:59.257139 19867 webserver.cc:460] Webserver started at http://127.0.98.3:42361/ using document root <none> and password file <none>
I20260925 14:52:59.257388 19867 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.257457 19867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.257601 19867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.258759 19867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "006d17a4b01e4f478da1c049e6dd071e"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "9693b8feca0b6069a741adec177d5d9f"
server_key_iv: "705de33889c35393832ff6d3278be43e"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.259126 19867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "006d17a4b01e4f478da1c049e6dd071e"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "9693b8feca0b6069a741adec177d5d9f"
server_key_iv: "705de33889c35393832ff6d3278be43e"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.260586 19867 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:59.261384 19882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.261572 19867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:59.261677 19867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "006d17a4b01e4f478da1c049e6dd071e"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "9693b8feca0b6069a741adec177d5d9f"
server_key_iv: "705de33889c35393832ff6d3278be43e"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.261782 19867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.276306 19867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.278510 19867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.278671 19867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.278936 19867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:59.279286 19867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:59.279341 19867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.279383 19867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:59.279412 19867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.285506 19867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41345
I20260925 14:52:59.285559 19995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41345 every 8 connection(s)
I20260925 14:52:59.285935 19867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:52:59.286883 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 19867
I20260925 14:52:59.286993 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:52:59.287161 392 external_mini_cluster.cc:1428] Setting key bcb992d4e0214a438d6b87c63d5777b5
Sep 25 14:52:59 dist-test-slave-z2s5 krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:52:59.295816 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.286759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47377 (local address 127.0.98.62:46795)
0925 14:52:59.286871 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.286874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.287069 (+ 195us) server_negotiation.cc:407] Connection header received
0925 14:52:59.288107 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.288108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.288121 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.288169 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.288817 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.289405 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.290435 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.290640 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.290712 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.292858 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.292859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.292859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.292867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.294363 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.294842 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.294844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.294844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.294911 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.295172 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.295172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.295173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.295293 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.295369 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.295647 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.295726 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:52:59.296010 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.286726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46795 (local address 127.0.98.3:47377)
0925 14:52:59.286932 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.286982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.287996 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.288240 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.288244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.288515 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.288712 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.288717 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.289496 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.289497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.290316 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.290318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.290730 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.290731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.290854 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.291328 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:52:59.291336 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.292746 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.294498 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.294505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.294507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.294754 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.294979 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:52:59.294980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:52:59.294980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:52:59.295092 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:52:59.295359 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.295365 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.295506 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.295617 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.295774 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":112,"threads_started":1}
I20260925 14:52:59.296489 19996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46795
I20260925 14:52:59.296586 19996 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:59.296773 19996 heartbeater.cc:511] Master 127.0.98.62:46795 requested a full tablet report, sending...
I20260925 14:52:59.297140 19537 ts_manager.cc:194] Registered new tserver with Master: 006d17a4b01e4f478da1c049e6dd071e (127.0.98.3:41345)
I20260925 14:52:59.297514 19537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47377
I20260925 14:52:59.298815 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:52:59.306151 19586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.300164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.98.62:46795)
0925 14:52:59.300210 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.300212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.300274 (+ 62us) server_negotiation.cc:407] Connection header received
0925 14:52:59.300350 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.300351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.300361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.300409 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.300982 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.301543 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.302300 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.302569 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.302655 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.303717 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.303718 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.303719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:52:59.303726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.304931 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.305311 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.305313 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.305313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.305368 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:52:59.305579 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:52:59.305580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:52:59.305581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:52:59.305740 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.305844 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.305979 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.306055 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:52:59.307767 19537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:52:59.309379 19537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:52:59.315508 20004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.311676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35125 (local address 127.0.0.1:59748)
0925 14:52:59.311917 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.311929 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.311979 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.312416 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.312417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.312419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:59.312497 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.312500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.313427 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.313428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.314455 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.314458 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.315178 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.315179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.315255 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.315269 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.315344 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.315432 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":73,"threads_started":1}
I20260925 14:52:59.315521 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.311820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.0.98.1:35125)
0925 14:52:59.312248 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.312273 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.312281 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:52:59.312312 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.312314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.312326 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.312418 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:59.312584 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.313316 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.314547 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.315083 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.315178 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.315241 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.315371 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.315432 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":86,"threads_started":1}
I20260925 14:52:59.316587 19662 tablet_service.cc:1467] Processing CreateTablet for tablet 63550217145b4bedbc235eab7f8cca6b (DEFAULT_TABLE table=test-table [id=1ab0e9f6be914a92b204e86b7653123f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:59.316728 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.311751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.0.98.3:41345)
0925 14:52:59.312040 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.312045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.312056 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:52:59.312093 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.312095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.312113 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.312239 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:59.312629 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.313861 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.315535 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.316332 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.316413 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.316467 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.316575 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.316626 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":80,"threads_started":1}
I20260925 14:52:59.316767 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.311880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34283 (local address 127.0.0.1:38166)
0925 14:52:59.312078 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.312087 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.312126 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.312523 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.312523 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.312525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:59.312557 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.312558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.313990 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.313991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.315424 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.315426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.316506 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.316507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.316579 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.316584 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.316635 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.316681 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":41,"thread_start_us":50,"threads_started":1}
I20260925 14:52:59.316869 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.311655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41345 (local address 127.0.0.1:33868)
0925 14:52:59.311841 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.311887 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.311958 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.312350 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.312354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.312356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:59.312535 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.312538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.314002 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.314003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.315423 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.315426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.316424 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.316425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.316489 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.316493 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.316549 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.316595 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":73,"thread_start_us":61,"threads_started":1}
I20260925 14:52:59.316965 19662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63550217145b4bedbc235eab7f8cca6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.317191 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.312068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.0.98.2:34283)
0925 14:52:59.312273 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.312277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.312288 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:52:59.312328 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.312331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.312347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.312428 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:59.312644 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.313852 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.315535 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.316334 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.316411 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.316467 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.317042 (+ 575us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.317093 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":83,"threads_started":1}
I20260925 14:52:59.317538 19930 tablet_service.cc:1467] Processing CreateTablet for tablet 63550217145b4bedbc235eab7f8cca6b (DEFAULT_TABLE table=test-table [id=1ab0e9f6be914a92b204e86b7653123f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:59.317857 19930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63550217145b4bedbc235eab7f8cca6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.317790 19796 tablet_service.cc:1467] Processing CreateTablet for tablet 63550217145b4bedbc235eab7f8cca6b (DEFAULT_TABLE table=test-table [id=1ab0e9f6be914a92b204e86b7653123f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:52:59.318023 19796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63550217145b4bedbc235eab7f8cca6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.320416 20010 tablet_bootstrap.cc:492] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: Bootstrap starting.
I20260925 14:52:59.321056 20011 tablet_bootstrap.cc:492] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Bootstrap starting.
I20260925 14:52:59.321213 20010 tablet_bootstrap.cc:654] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:59.321355 20012 tablet_bootstrap.cc:492] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Bootstrap starting.
I20260925 14:52:59.321765 20011 tablet_bootstrap.cc:654] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:59.321914 20010 log.cc:824] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:59.322081 20012 tablet_bootstrap.cc:654] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:59.322374 20011 log.cc:824] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:59.322721 20012 log.cc:824] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:59.322839 20010 tablet_bootstrap.cc:492] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: No bootstrap required, opened a new log
I20260925 14:52:59.323063 20010 ts_tablet_manager.cc:1397] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:52:59.323279 20011 tablet_bootstrap.cc:492] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: No bootstrap required, opened a new log
I20260925 14:52:59.323350 20011 ts_tablet_manager.cc:1397] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:59.323518 20012 tablet_bootstrap.cc:492] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: No bootstrap required, opened a new log
I20260925 14:52:59.323583 20012 ts_tablet_manager.cc:1397] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:52:59.324880 20010 raft_consensus.cc:348] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.325060 20010 raft_consensus.cc:374] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:59.325129 20010 raft_consensus.cc:729] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d29297f7f0141bfb8a89b19008f5d77, State: Initialized, Role: FOLLOWER
I20260925 14:52:59.325268 20010 consensus_queue.cc:260] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.325301 20011 raft_consensus.cc:348] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.325531 20011 raft_consensus.cc:374] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:59.325592 20011 raft_consensus.cc:729] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006d17a4b01e4f478da1c049e6dd071e, State: Initialized, Role: FOLLOWER
I20260925 14:52:59.325610 20010 ts_tablet_manager.cc:1428] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:52:59.325541 20012 raft_consensus.cc:348] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.325693 20012 raft_consensus.cc:374] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:59.325702 20011 consensus_queue.cc:260] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.325757 19728 heartbeater.cc:503] Master 127.0.98.62:46795 was elected leader, sending a full tablet report...
I20260925 14:52:59.325773 20012 raft_consensus.cc:729] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b87620ac134b70bcf6443290ec1b98, State: Initialized, Role: FOLLOWER
I20260925 14:52:59.325909 20012 consensus_queue.cc:260] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.325997 20011 ts_tablet_manager.cc:1428] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:52:59.326031 19996 heartbeater.cc:503] Master 127.0.98.62:46795 was elected leader, sending a full tablet report...
I20260925 14:52:59.326179 20012 ts_tablet_manager.cc:1428] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 14:52:59.326491 19862 heartbeater.cc:503] Master 127.0.98.62:46795 was elected leader, sending a full tablet report...
W20260925 14:52:59.380213 19863 tablet.cc:2369] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:52:59.467694 19729 tablet.cc:2369] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:52:59.470904 20017 raft_consensus.cc:482] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:52:59.471043 20017 raft_consensus.cc:504] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.471396 20017 leader_election.cc:288] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d29297f7f0141bfb8a89b19008f5d77 (127.0.98.1:35125), 64b87620ac134b70bcf6443290ec1b98 (127.0.98.2:34283)
I20260925 14:52:59.474370 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.471622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47141 (local address 127.0.98.1:35125)
0925 14:52:59.471663 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.471666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.471777 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:52:59.471819 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.471820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.471835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.471890 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:59.472083 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.472590 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.473622 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.474005 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.474059 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.474068 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.474256 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.474297 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:52:59.474426 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.471757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39303 (local address 127.0.98.2:34283)
0925 14:52:59.471817 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.471820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.471829 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:52:59.471903 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.471904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.471916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.471978 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:52:59.472164 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.472604 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.473660 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.474021 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.474113 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.474123 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.474285 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.474325 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:52:59.474385 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.471598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35125 (local address 127.0.98.3:47141)
0925 14:52:59.471657 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.471669 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.471735 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.471911 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.471912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.471915 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:59.471988 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.471990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.472687 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.472689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.473521 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.473524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.474091 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.474092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.474178 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.474181 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.474223 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.474289 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":30}
I20260925 14:52:59.474413 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.471609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34283 (local address 127.0.98.3:39303)
0925 14:52:59.471763 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.471774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.471826 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.471987 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.471987 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.471988 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:52:59.472086 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.472087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.472715 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.472716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.473537 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.473539 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.474135 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.474135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.474188 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.474191 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.474263 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.474300 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":28,"thread_start_us":95,"threads_started":1}
I20260925 14:52:59.474819 19816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63550217145b4bedbc235eab7f8cca6b" candidate_uuid: "006d17a4b01e4f478da1c049e6dd071e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b87620ac134b70bcf6443290ec1b98" is_pre_election: true
I20260925 14:52:59.474817 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63550217145b4bedbc235eab7f8cca6b" candidate_uuid: "006d17a4b01e4f478da1c049e6dd071e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d29297f7f0141bfb8a89b19008f5d77" is_pre_election: true
I20260925 14:52:59.474985 19682 raft_consensus.cc:2454] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 006d17a4b01e4f478da1c049e6dd071e in term 0.
I20260925 14:52:59.474985 19816 raft_consensus.cc:2454] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 006d17a4b01e4f478da1c049e6dd071e in term 0.
I20260925 14:52:59.475225 19885 leader_election.cc:302] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006d17a4b01e4f478da1c049e6dd071e, 64b87620ac134b70bcf6443290ec1b98; no voters:
I20260925 14:52:59.475379 20017 raft_consensus.cc:2781] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:52:59.475435 20017 raft_consensus.cc:482] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:52:59.475468 20017 raft_consensus.cc:3037] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:59.476182 20017 raft_consensus.cc:504] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.476333 20017 leader_election.cc:288] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [CANDIDATE]: Term 1 election: Requested vote from peers 4d29297f7f0141bfb8a89b19008f5d77 (127.0.98.1:35125), 64b87620ac134b70bcf6443290ec1b98 (127.0.98.2:34283)
I20260925 14:52:59.476521 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63550217145b4bedbc235eab7f8cca6b" candidate_uuid: "006d17a4b01e4f478da1c049e6dd071e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d29297f7f0141bfb8a89b19008f5d77"
I20260925 14:52:59.476532 19816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63550217145b4bedbc235eab7f8cca6b" candidate_uuid: "006d17a4b01e4f478da1c049e6dd071e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b87620ac134b70bcf6443290ec1b98"
I20260925 14:52:59.476598 19816 raft_consensus.cc:3037] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:59.476634 19682 raft_consensus.cc:3037] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:59.477315 19682 raft_consensus.cc:2454] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 006d17a4b01e4f478da1c049e6dd071e in term 1.
I20260925 14:52:59.477370 19816 raft_consensus.cc:2454] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 006d17a4b01e4f478da1c049e6dd071e in term 1.
I20260925 14:52:59.477548 19884 leader_election.cc:302] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006d17a4b01e4f478da1c049e6dd071e, 4d29297f7f0141bfb8a89b19008f5d77; no voters:
I20260925 14:52:59.477661 20017 raft_consensus.cc:2781] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:59.477836 20017 raft_consensus.cc:686] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 1 LEADER]: Becoming Leader. State: Replica: 006d17a4b01e4f478da1c049e6dd071e, State: Running, Role: LEADER
I20260925 14:52:59.477954 20017 consensus_queue.cc:237] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } }
I20260925 14:52:59.478919 19536 catalog_manager.cc:5495] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e reported cstate change: term changed from 0 to 1, leader changed from <none> to 006d17a4b01e4f478da1c049e6dd071e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "006d17a4b01e4f478da1c049e6dd071e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "006d17a4b01e4f478da1c049e6dd071e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 } health_report { overall_health: UNKNOWN } } }
I20260925 14:52:59.496870 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.494585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.0.98.3:41345)
0925 14:52:59.494654 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.494656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.494665 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:52:59.494732 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.494733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.494750 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.494816 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:52:59.494928 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.495390 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.496165 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.496346 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.496395 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.496529 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:52:59.496625 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:52:59.496687 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.496776 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.496793 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:52:59.498744 19816 raft_consensus.cc:1261] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 1 FOLLOWER]: Refusing update from remote peer 006d17a4b01e4f478da1c049e6dd071e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:59.498766 19682 raft_consensus.cc:1261] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 1 FOLLOWER]: Refusing update from remote peer 006d17a4b01e4f478da1c049e6dd071e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:52:59.499091 20020 consensus_queue.cc:1035] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b87620ac134b70bcf6443290ec1b98" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:59.499274 20017 consensus_queue.cc:1035] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d29297f7f0141bfb8a89b19008f5d77" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:52:59.501703 20023 mvcc.cc:205] Tried to move back new op lower bound from 7333265324023689216 to 7333265323942318080. Current Snapshot: MvccSnapshot[applied={T|T < 7333265324023689216}]
I20260925 14:52:59.501876 20024 mvcc.cc:205] Tried to move back new op lower bound from 7333265324023689216 to 7333265323942318080. Current Snapshot: MvccSnapshot[applied={T|T < 7333265324023689216}]
I20260925 14:52:59.502044 20025 mvcc.cc:205] Tried to move back new op lower bound from 7333265324023689216 to 7333265323942318080. Current Snapshot: MvccSnapshot[applied={T|T < 7333265324023689216}]
I20260925 14:52:59.504683 19537 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:59.504786 19537 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:52:59.505555 19537 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1557c45dc193489f988b44dbb25c001c: Sending DeleteTablet for 3 replicas of tablet 63550217145b4bedbc235eab7f8cca6b
I20260925 14:52:59.505890 19662 tablet_service.cc:1514] Processing DeleteTablet for tablet 63550217145b4bedbc235eab7f8cca6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59748
I20260925 14:52:59.505954 19930 tablet_service.cc:1514] Processing DeleteTablet for tablet 63550217145b4bedbc235eab7f8cca6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33868
I20260925 14:52:59.506160 20033 tablet_replica.cc:330] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77: stopping tablet replica
I20260925 14:52:59.506177 19796 tablet_service.cc:1514] Processing DeleteTablet for tablet 63550217145b4bedbc235eab7f8cca6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:52:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38166
I20260925 14:52:59.506261 20033 raft_consensus.cc:2229] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:59.506359 20033 raft_consensus.cc:2258] T 63550217145b4bedbc235eab7f8cca6b P 4d29297f7f0141bfb8a89b19008f5d77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:59.506366 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19599
I20260925 14:52:59.506439 20035 tablet_replica.cc:330] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: stopping tablet replica
I20260925 14:52:59.506547 20035 raft_consensus.cc:2229] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:52:59.506659 20035 raft_consensus.cc:2258] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:59.507248 20035 ts_tablet_manager.cc:1905] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:59.509284 20035 ts_tablet_manager.cc:1918] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:59.509378 20035 log.cc:1196] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/63550217145b4bedbc235eab7f8cca6b
I20260925 14:52:59.509666 20035 ts_tablet_manager.cc:1939] T 63550217145b4bedbc235eab7f8cca6b P 64b87620ac134b70bcf6443290ec1b98: Deleting consensus metadata
I20260925 14:52:59.510432 19523 catalog_manager.cc:4841] TS 64b87620ac134b70bcf6443290ec1b98 (127.0.98.2:34283): tablet 63550217145b4bedbc235eab7f8cca6b (table test-table [id=1ab0e9f6be914a92b204e86b7653123f]) successfully deleted
I20260925 14:52:59.510605 20034 tablet_replica.cc:330] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: stopping tablet replica
I20260925 14:52:59.510705 20034 raft_consensus.cc:2229] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:52:59.510844 20034 raft_consensus.cc:2258] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:52:59.511310 20034 ts_tablet_manager.cc:1905] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:52:59.512929 20034 ts_tablet_manager.cc:1918] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:52:59.513000 20034 log.cc:1196] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/63550217145b4bedbc235eab7f8cca6b
I20260925 14:52:59.513309 20034 ts_tablet_manager.cc:1939] T 63550217145b4bedbc235eab7f8cca6b P 006d17a4b01e4f478da1c049e6dd071e: Deleting consensus metadata
I20260925 14:52:59.513700 19523 catalog_manager.cc:4841] TS 006d17a4b01e4f478da1c049e6dd071e (127.0.98.3:41345): tablet 63550217145b4bedbc235eab7f8cca6b (table test-table [id=1ab0e9f6be914a92b204e86b7653123f]) successfully deleted
W20260925 14:52:59.513878 19522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35125 (error 108)
I20260925 14:52:59.514164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19733
I20260925 14:52:59.514333 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.514221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35125 (local address 127.0.0.1:59760)
0925 14:52:59.514271 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.514304 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:52:59.514909 19522 catalog_manager.cc:4584] TS 4d29297f7f0141bfb8a89b19008f5d77 (127.0.98.1:35125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63550217145b4bedbc235eab7f8cca6b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35125: connect: Connection refused (error 111)
I20260925 14:52:59.520133 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19867
I20260925 14:52:59.525038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 19505
2026-09-25T14:52:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2041 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T14:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:52:59Z Disabled control of system clock
I20260925 14:52:59.552556 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35387
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:59.634232 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.634569 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.634596 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.635787 20040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:52:59.635828 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.635840 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.635854 20040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:52:59.635865 20040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:52:59.637156 20040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:59.637369 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.637630 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.642496 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.642558 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.642743 20040 server_base.cc:1034] running on GCE node
W20260925 14:52:59.642596 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.643051 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.643314 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.644486 20040 hybrid_clock.cc:648] HybridClock initialized: now 1790347979644470 us; error 44 us; skew 500 ppm
I20260925 14:52:59.645901 20040 webserver.cc:460] Webserver started at http://127.0.98.62:42477/ using document root <none> and password file <none>
I20260925 14:52:59.646160 20040 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.646234 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.646426 20040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.647526 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "0c0cfd9da54644b88bc43853e17bfc04"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "61b908ba069c3217c7234065499f1cda"
server_key_iv: "488b051a92a869bf8354075d30529d35"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.647902 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c0cfd9da54644b88bc43853e17bfc04"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "61b908ba069c3217c7234065499f1cda"
server_key_iv: "488b051a92a869bf8354075d30529d35"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.649303 20040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:52:59.650072 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.650246 20040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:59.650354 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "0c0cfd9da54644b88bc43853e17bfc04"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "61b908ba069c3217c7234065499f1cda"
server_key_iv: "488b051a92a869bf8354075d30529d35"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.650471 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.659017 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.659341 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.659493 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.663643 20040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35387
I20260925 14:52:59.663695 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35387 every 8 connection(s)
I20260925 14:52:59.664062 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:52:59.665071 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:52:59.666837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20040
I20260925 14:52:59.666958 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:52:59.667155 392 external_mini_cluster.cc:1428] Setting key 4b9322902cb6183ded096a4f63b536f0
I20260925 14:52:59.668138 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04: Bootstrap starting.
I20260925 14:52:59.668917 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04: Neither blocks nor log segments found. Creating new log.
I20260925 14:52:59.669282 20107 log.cc:824] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04: Log is configured to *not* fsync() on all Append() calls
I20260925 14:52:59.670080 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04: No bootstrap required, opened a new log
I20260925 14:52:59.672555 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0cfd9da54644b88bc43853e17bfc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35387 } }
I20260925 14:52:59.672742 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:52:59.672794 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c0cfd9da54644b88bc43853e17bfc04, State: Initialized, Role: FOLLOWER
I20260925 14:52:59.672966 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0cfd9da54644b88bc43853e17bfc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35387 } }
I20260925 14:52:59.673074 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:52:59.673112 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:52:59.673170 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:52:59.673730 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.667774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.0.98.62:35387)
0925 14:52:59.668020 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.668030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.668066 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:52:59.668701 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.668711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.668807 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.668983 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.669203 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.670569 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.671343 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.672045 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.672230 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.672889 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.672893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.672893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:59.673132 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.673182 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:59.673199 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.673275 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.673404 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.673492 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260925 14:52:59.674003 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0cfd9da54644b88bc43853e17bfc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35387 } }
I20260925 14:52:59.674132 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c0cfd9da54644b88bc43853e17bfc04; no voters:
I20260925 14:52:59.674309 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:52:59.674365 20112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:52:59.674558 20112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [term 1 LEADER]: Becoming Leader. State: Replica: 0c0cfd9da54644b88bc43853e17bfc04, State: Running, Role: LEADER
I20260925 14:52:59.674618 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:52:59.674654 20112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0cfd9da54644b88bc43853e17bfc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35387 } }
I20260925 14:52:59.674980 20112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c0cfd9da54644b88bc43853e17bfc04. Latest consensus state: current_term: 1 leader_uuid: "0c0cfd9da54644b88bc43853e17bfc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0cfd9da54644b88bc43853e17bfc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35387 } } }
I20260925 14:52:59.674991 20114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c0cfd9da54644b88bc43853e17bfc04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c0cfd9da54644b88bc43853e17bfc04" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35387 } } }
I20260925 14:52:59.675104 20112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:59.675172 20114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04 [sys.catalog]: This master's current role is: LEADER
I20260925 14:52:59.676219 20118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:52:59.676791 20118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:52:59.678409 20118 catalog_manager.cc:1347] Generated new cluster ID: 920af0c5f94548a483492d3e386171a5
I20260925 14:52:59.678467 20118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:52:59.686172 20118 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:52:59.686965 20118 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:52:59.704967 20118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04: Generated new TSK 0
I20260925 14:52:59.705206 20118 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:52:59.714001 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35387
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:59.796561 20131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.796810 20131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.796849 20131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.797996 20131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.798072 20131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.798115 20131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260925 14:52:59.799546 20131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:59.799832 20131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.800141 20131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.806341 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.806341 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.806450 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.806870 20131 server_base.cc:1034] running on GCE node
I20260925 14:52:59.807106 20131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.807389 20131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.808555 20131 hybrid_clock.cc:648] HybridClock initialized: now 1790347979808531 us; error 44 us; skew 500 ppm
I20260925 14:52:59.809747 20131 webserver.cc:460] Webserver started at http://127.0.98.1:35165/ using document root <none> and password file <none>
I20260925 14:52:59.810014 20131 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.810083 20131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.810248 20131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.811404 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc95658679904c3d8a879c794c6be327"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "d8b18d289be110540dac9dd7dc29f395"
server_key_iv: "5e63cccffb6d8140f044f34ec4f28402"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.811764 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc95658679904c3d8a879c794c6be327"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "d8b18d289be110540dac9dd7dc29f395"
server_key_iv: "5e63cccffb6d8140f044f34ec4f28402"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.813046 20131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:52:59.813800 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.813973 20131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:52:59.814069 20131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal
uuid: "cc95658679904c3d8a879c794c6be327"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "d8b18d289be110540dac9dd7dc29f395"
server_key_iv: "5e63cccffb6d8140f044f34ec4f28402"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.814162 20131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.834638 20131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.834990 20131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.835161 20131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.835407 20131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:59.835772 20131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:59.835834 20131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.835873 20131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:59.835901 20131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.841725 20131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40785
I20260925 14:52:59.841775 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40785 every 8 connection(s)
I20260925 14:52:59.842105 20131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/data/info.pb
I20260925 14:52:59.847524 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.842865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35185 (local address 127.0.98.62:35387)
0925 14:52:59.842936 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.842938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.843177 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:52:59.843747 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.843748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.843760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.843815 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.844493 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.845111 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.846095 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.846327 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.846421 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.847226 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.847227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.847228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:59.847233 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.847249 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:59.847253 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.847307 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.847402 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.847437 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:52:59.847738 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.842862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35387 (local address 127.0.98.1:35185)
0925 14:52:59.843044 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.843079 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.843590 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.843890 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.843893 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.843901 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.844399 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.844404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.845207 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.845209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.845989 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.845991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.846434 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.846435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.846564 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.847044 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:59.847044 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.847072 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:59.847095 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.847310 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.847315 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.847319 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.847405 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.847510 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260925 14:52:59.848238 20259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35387
I20260925 14:52:59.848368 20259 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:59.848569 20259 heartbeater.cc:511] Master 127.0.98.62:35387 requested a full tablet report, sending...
I20260925 14:52:59.849094 20071 ts_manager.cc:194] Registered new tserver with Master: cc95658679904c3d8a879c794c6be327 (127.0.98.1:40785)
I20260925 14:52:59.849386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20131
I20260925 14:52:59.849469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-0/wal/instance
I20260925 14:52:59.849674 392 external_mini_cluster.cc:1428] Setting key f29ba702b1cb3a7e2786b7fdf603d9bf
I20260925 14:52:59.849803 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35185
I20260925 14:52:59.850730 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35387
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 14:52:59.931942 20262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:52:59.932149 20262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:52:59.932173 20262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:52:59.933342 20262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:52:59.933386 20262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:52:59.933411 20262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260925 14:52:59.934842 20262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:52:59.935081 20262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:52:59.935403 20262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:52:59.940441 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.940459 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:52:59.940443 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:52:59.940712 20262 server_base.cc:1034] running on GCE node
I20260925 14:52:59.940934 20262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:52:59.941177 20262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:52:59.942337 20262 hybrid_clock.cc:648] HybridClock initialized: now 1790347979942334 us; error 38 us; skew 500 ppm
I20260925 14:52:59.943703 20262 webserver.cc:460] Webserver started at http://127.0.98.2:33479/ using document root <none> and password file <none>
I20260925 14:52:59.943967 20262 fs_manager.cc:359] Metadata directory not provided
I20260925 14:52:59.944039 20262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:52:59.944188 20262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:52:59.945307 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data/instance:
uuid: "2fe5f253d13c40e08f16b2c46b8cd619"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "a30c9918edd8905bd014a19784a71e89"
server_key_iv: "0df92ee88aed8fb4ba4869fb5eb16a95"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.945657 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2fe5f253d13c40e08f16b2c46b8cd619"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "a30c9918edd8905bd014a19784a71e89"
server_key_iv: "0df92ee88aed8fb4ba4869fb5eb16a95"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.947045 20262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:52:59.947824 20276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.948019 20262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:52:59.948120 20262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal
uuid: "2fe5f253d13c40e08f16b2c46b8cd619"
format_stamp: "Formatted at 2026-09-25 14:52:59 on dist-test-slave-z2s5"
server_key: "a30c9918edd8905bd014a19784a71e89"
server_key_iv: "0df92ee88aed8fb4ba4869fb5eb16a95"
server_key_version: "encryptionkey@0"
I20260925 14:52:59.948194 20262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:52:59.956936 20262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:52:59.957247 20262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:52:59.957436 20262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:52:59.957655 20262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:52:59.957978 20262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:52:59.958031 20262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.958072 20262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:52:59.958101 20262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:52:59.964102 20262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44445
I20260925 14:52:59.964147 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44445 every 8 connection(s)
I20260925 14:52:59.964489 20262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/data/info.pb
I20260925 14:52:59.965523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20262
I20260925 14:52:59.965654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal/instance
I20260925 14:52:59.965868 392 external_mini_cluster.cc:1428] Setting key 8926b332c7f2ba71fa3e8bbdae8d34a3
I20260925 14:52:59.966943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35387
--builtin_ntp_servers=127.0.98.20:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 14:52:59.970229 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.965217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53439 (local address 127.0.98.62:35387)
0925 14:52:59.965269 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:52:59.965273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:52:59.965493 (+ 220us) server_negotiation.cc:407] Connection header received
0925 14:52:59.966176 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:52:59.966177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:52:59.966190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:52:59.966261 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:52:59.967151 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.967649 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.968647 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.968986 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.969065 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.969985 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:52:59.969986 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:52:59.969987 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:52:59.969991 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:52:59.970004 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:52:59.970008 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:52:59.970057 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:52:59.970127 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:52:59.970154 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:52:59.970510 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:52:59.965180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35387 (local address 127.0.98.2:53439)
0925 14:52:59.965367 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:52:59.965402 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:52:59.966042 (+ 640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:52:59.966352 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:52:59.966357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:52:59.966368 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:52:59.967018 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.967024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.967745 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.967745 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.968557 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:52:59.968559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:52:59.969078 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:52:59.969079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:52:59.969197 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:52:59.969818 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:52:59.969818 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:52:59.969848 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:52:59.969873 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:52:59.970057 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:52:59.970062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:52:59.970065 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:52:59.970133 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:52:59.970236 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260925 14:52:59.970984 20390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35387
I20260925 14:52:59.971100 20390 heartbeater.cc:464] Registering TS with master...
I20260925 14:52:59.971328 20390 heartbeater.cc:511] Master 127.0.98.62:35387 requested a full tablet report, sending...
I20260925 14:52:59.971724 20070 ts_manager.cc:194] Registered new tserver with Master: 2fe5f253d13c40e08f16b2c46b8cd619 (127.0.98.2:44445)
I20260925 14:52:59.972110 20070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:53439
W20260925 14:53:00.049697 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:00.049937 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:00.049959 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:00.051251 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:00.051326 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:00.051352 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260925 14:53:00.052906 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:00.053112 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:00.053364 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:00.058485 20401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:00.058480 20399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:00.058471 20398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:00.058728 20393 server_base.cc:1034] running on GCE node
I20260925 14:53:00.059029 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:00.059299 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:00.060441 20393 hybrid_clock.cc:648] HybridClock initialized: now 1790347980060427 us; error 30 us; skew 500 ppm
I20260925 14:53:00.061822 20393 webserver.cc:460] Webserver started at http://127.0.98.3:42483/ using document root <none> and password file <none>
I20260925 14:53:00.062076 20393 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:00.062151 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:00.062302 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:00.063417 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data/instance:
uuid: "35e306cdd88f45e68452f62bdc655980"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-z2s5"
server_key: "343667d9880dea0cd4a967764026620d"
server_key_iv: "c49a6f54038e624d74d27945c2d92a6d"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.063786 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal/instance:
uuid: "35e306cdd88f45e68452f62bdc655980"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-z2s5"
server_key: "343667d9880dea0cd4a967764026620d"
server_key_iv: "c49a6f54038e624d74d27945c2d92a6d"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.065093 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:00.065938 20407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.066267 20393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:00.066376 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal
uuid: "35e306cdd88f45e68452f62bdc655980"
format_stamp: "Formatted at 2026-09-25 14:53:00 on dist-test-slave-z2s5"
server_key: "343667d9880dea0cd4a967764026620d"
server_key_iv: "c49a6f54038e624d74d27945c2d92a6d"
server_key_version: "encryptionkey@0"
I20260925 14:53:00.066505 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:00.083433 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:00.083796 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:00.083956 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:00.084199 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:53:00.084549 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:53:00.084604 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.084646 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:53:00.084676 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:00.090255 20393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36097
I20260925 14:53:00.090332 20520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36097 every 8 connection(s)
I20260925 14:53:00.090701 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/data/info.pb
I20260925 14:53:00.092226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20393
I20260925 14:53:00.092332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal/instance
I20260925 14:53:00.092551 392 external_mini_cluster.cc:1428] Setting key 1e1c4df3a227c026fe834d5c6a0c4827
I20260925 14:53:00.096668 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.091562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52811 (local address 127.0.98.62:35387)
0925 14:53:00.091612 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.091614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.091856 (+ 242us) server_negotiation.cc:407] Connection header received
0925 14:53:00.092563 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.092564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.092576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.092715 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.093716 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.094211 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.095355 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.095597 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.095649 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.096458 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.096460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.096460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:00.096464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.096476 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:00.096479 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.096521 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.096586 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.096608 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:53:00.096974 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.091525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35387 (local address 127.0.98.3:52811)
0925 14:53:00.091724 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.091766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.092429 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.092792 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.092796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.092807 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:00.093522 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.093529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.094302 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.094303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.095254 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.095257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.095675 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.095676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.095811 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.096326 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:53:00.096326 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:00.096355 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:53:00.096378 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:00.096523 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:00.096526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:00.096528 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.096581 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.096737 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260925 14:53:00.097473 20521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35387
I20260925 14:53:00.097581 20521 heartbeater.cc:464] Registering TS with master...
I20260925 14:53:00.097851 20521 heartbeater.cc:511] Master 127.0.98.62:35387 requested a full tablet report, sending...
I20260925 14:53:00.098238 20070 ts_manager.cc:194] Registered new tserver with Master: 35e306cdd88f45e68452f62bdc655980 (127.0.98.3:36097)
I20260925 14:53:00.098650 20070 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52811
I20260925 14:53:00.104689 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:53:00.109161 20110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.106136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36848 (local address 127.0.98.62:35387)
0925 14:53:00.106219 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.106221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.106310 (+ 89us) server_negotiation.cc:407] Connection header received
0925 14:53:00.106383 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.106384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.106413 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.106465 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:00.106639 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.107123 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.107917 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.108106 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.108181 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.108896 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:00.108897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:00.108898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:00.108902 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:00.108916 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:00.108920 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:00.108982 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.109050 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.109072 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 14:53:00.110832 20070 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:53:00.112417 20070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:53:00.119187 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.114878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.0.98.2:44445)
0925 14:53:00.115343 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.115349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.115360 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:00.115408 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.115411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.115428 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.115510 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:00.115778 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.116951 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.118206 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.118790 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.118878 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.118974 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.119056 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.119095 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":125,"threads_started":1}
I20260925 14:53:00.119288 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.115021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40785 (local address 127.0.0.1:45360)
0925 14:53:00.115238 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.115249 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.115306 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.115590 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.115593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.115598 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:00.115687 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.115689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.116725 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.116726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.118107 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.118109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.119042 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.119043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.119100 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.119104 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.119153 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.119194 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":10,"thread_start_us":76,"threads_started":1}
I20260925 14:53:00.119175 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.114785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44445 (local address 127.0.0.1:42166)
0925 14:53:00.115179 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.115215 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.115284 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.115590 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.115593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.115595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:00.115687 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.115690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.117031 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.117031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.118106 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.118108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.118877 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.118879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.118952 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.118964 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.119041 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.119095 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":74,"spinlock_wait_cycles":1280,"thread_start_us":82,"threads_started":1}
I20260925 14:53:00.119484 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.115121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45360 (local address 127.0.98.1:40785)
0925 14:53:00.115427 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.115431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.115439 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:00.115475 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.115477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.115492 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.115780 (+ 288us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:00.115844 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.116643 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.118205 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.118954 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.119240 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.119339 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.119363 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.119402 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":80,"threads_started":1}
I20260925 14:53:00.120041 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.114785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36097 (local address 127.0.0.1:36750)
0925 14:53:00.115172 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.115272 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.115320 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.115614 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.115615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.115617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:00.115684 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.115685 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.117852 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.117853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.118920 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.118922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.119833 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.119834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.119891 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.119893 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.119935 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.119973 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":81,"threads_started":1}
I20260925 14:53:00.120056 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.114986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36750 (local address 127.0.98.3:36097)
0925 14:53:00.115175 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.115182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.115433 (+ 251us) server_negotiation.cc:407] Connection header received
0925 14:53:00.115479 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.115482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.115499 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.115570 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:00.115841 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.116951 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.119024 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.119756 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.119833 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.119914 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.119945 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.119979 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":100,"threads_started":1}
I20260925 14:53:00.120151 20193 tablet_service.cc:1467] Processing CreateTablet for tablet f07efbb9c17947ac8d151f1ea2bcf459 (DEFAULT_TABLE table=test-table [id=9ab406d5bdf04eb48d2eb786bd7cb91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:00.120497 20193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07efbb9c17947ac8d151f1ea2bcf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:00.120687 20455 tablet_service.cc:1467] Processing CreateTablet for tablet f07efbb9c17947ac8d151f1ea2bcf459 (DEFAULT_TABLE table=test-table [id=9ab406d5bdf04eb48d2eb786bd7cb91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:00.120930 20455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07efbb9c17947ac8d151f1ea2bcf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:00.123149 20535 tablet_bootstrap.cc:492] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: Bootstrap starting.
I20260925 14:53:00.123519 20536 tablet_bootstrap.cc:492] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Bootstrap starting.
I20260925 14:53:00.123878 20535 tablet_bootstrap.cc:654] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:00.124089 20536 tablet_bootstrap.cc:654] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:00.124209 20535 log.cc:824] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:00.124405 20536 log.cc:824] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:00.124683 20324 tablet_service.cc:1467] Processing CreateTablet for tablet f07efbb9c17947ac8d151f1ea2bcf459 (DEFAULT_TABLE table=test-table [id=9ab406d5bdf04eb48d2eb786bd7cb91a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:53:00.124971 20535 tablet_bootstrap.cc:492] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: No bootstrap required, opened a new log
I20260925 14:53:00.125003 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07efbb9c17947ac8d151f1ea2bcf459. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:00.125038 20535 ts_tablet_manager.cc:1397] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:00.127111 20535 raft_consensus.cc:348] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.127243 20535 raft_consensus.cc:374] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:00.127266 20535 raft_consensus.cc:729] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc95658679904c3d8a879c794c6be327, State: Initialized, Role: FOLLOWER
I20260925 14:53:00.127388 20535 consensus_queue.cc:260] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.127631 20539 tablet_bootstrap.cc:492] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Bootstrap starting.
I20260925 14:53:00.127671 20535 ts_tablet_manager.cc:1428] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:00.127813 20259 heartbeater.cc:503] Master 127.0.98.62:35387 was elected leader, sending a full tablet report...
I20260925 14:53:00.128314 20539 tablet_bootstrap.cc:654] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:00.128396 20536 tablet_bootstrap.cc:492] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: No bootstrap required, opened a new log
I20260925 14:53:00.128468 20536 ts_tablet_manager.cc:1397] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 14:53:00.128649 20539 log.cc:824] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:00.129343 20539 tablet_bootstrap.cc:492] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: No bootstrap required, opened a new log
I20260925 14:53:00.129383 20539 ts_tablet_manager.cc:1397] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:53:00.130688 20536 raft_consensus.cc:348] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.130805 20536 raft_consensus.cc:374] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:00.130824 20536 raft_consensus.cc:729] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35e306cdd88f45e68452f62bdc655980, State: Initialized, Role: FOLLOWER
I20260925 14:53:00.130904 20536 consensus_queue.cc:260] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.131084 20536 ts_tablet_manager.cc:1428] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:00.131196 20521 heartbeater.cc:503] Master 127.0.98.62:35387 was elected leader, sending a full tablet report...
I20260925 14:53:00.131580 20539 raft_consensus.cc:348] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.131744 20539 raft_consensus.cc:374] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:00.131803 20539 raft_consensus.cc:729] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe5f253d13c40e08f16b2c46b8cd619, State: Initialized, Role: FOLLOWER
I20260925 14:53:00.131948 20539 consensus_queue.cc:260] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.132207 20539 ts_tablet_manager.cc:1428] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:53:00.132287 20390 heartbeater.cc:503] Master 127.0.98.62:35387 was elected leader, sending a full tablet report...
W20260925 14:53:00.215022 20391 tablet.cc:2369] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:00.289815 20540 raft_consensus.cc:482] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:53:00.290005 20540 raft_consensus.cc:504] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.290421 20540 leader_election.cc:288] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fe5f253d13c40e08f16b2c46b8cd619 (127.0.98.2:44445), 35e306cdd88f45e68452f62bdc655980 (127.0.98.3:36097)
I20260925 14:53:00.293220 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.290632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36097 (local address 127.0.98.1:50647)
0925 14:53:00.290692 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.290704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.290752 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.290954 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.290955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.290957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:00.291030 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.291031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.291673 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.291674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.292410 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.292412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.292964 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.292965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.293031 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.293034 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.293107 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.293150 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:53:00.293220 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.290728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50647 (local address 127.0.98.3:36097)
0925 14:53:00.290824 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.290826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.290836 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:00.290870 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.290870 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.290878 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.290948 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:00.291110 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.291576 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.292504 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.292868 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.292951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.292961 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.293117 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.293150 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 14:53:00.293493 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.290641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44445 (local address 127.0.98.1:58669)
0925 14:53:00.290910 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.290919 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:00.291025 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:00.291215 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:00.291215 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:00.291217 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:53:00.291284 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.291285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.291945 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.291946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.292679 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:00.292680 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.293240 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.293241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:00.293310 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.293315 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:53:00.293374 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:53:00.293419 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":71,"thread_start_us":121,"threads_started":1}
I20260925 14:53:00.293498 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.290742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58669 (local address 127.0.98.2:44445)
0925 14:53:00.290792 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.290794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.291000 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:53:00.291114 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.291115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.291124 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.291212 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:53:00.291366 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.291855 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.292789 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.293151 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.293232 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.293241 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.293384 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.293432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:00.293560 20475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07efbb9c17947ac8d151f1ea2bcf459" candidate_uuid: "cc95658679904c3d8a879c794c6be327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e306cdd88f45e68452f62bdc655980" is_pre_election: true
I20260925 14:53:00.293737 20475 raft_consensus.cc:2454] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc95658679904c3d8a879c794c6be327 in term 0.
I20260925 14:53:00.293965 20344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07efbb9c17947ac8d151f1ea2bcf459" candidate_uuid: "cc95658679904c3d8a879c794c6be327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" is_pre_election: true
I20260925 14:53:00.294016 20149 leader_election.cc:302] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e306cdd88f45e68452f62bdc655980, cc95658679904c3d8a879c794c6be327; no voters:
I20260925 14:53:00.294112 20344 raft_consensus.cc:2454] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc95658679904c3d8a879c794c6be327 in term 0.
I20260925 14:53:00.294130 20540 raft_consensus.cc:2781] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:53:00.294183 20540 raft_consensus.cc:482] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:53:00.294246 20540 raft_consensus.cc:3037] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:00.295006 20540 raft_consensus.cc:504] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.295146 20540 leader_election.cc:288] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [CANDIDATE]: Term 1 election: Requested vote from peers 2fe5f253d13c40e08f16b2c46b8cd619 (127.0.98.2:44445), 35e306cdd88f45e68452f62bdc655980 (127.0.98.3:36097)
I20260925 14:53:00.295315 20344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07efbb9c17947ac8d151f1ea2bcf459" candidate_uuid: "cc95658679904c3d8a879c794c6be327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fe5f253d13c40e08f16b2c46b8cd619"
I20260925 14:53:00.295406 20344 raft_consensus.cc:3037] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:00.295441 20475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07efbb9c17947ac8d151f1ea2bcf459" candidate_uuid: "cc95658679904c3d8a879c794c6be327" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35e306cdd88f45e68452f62bdc655980"
I20260925 14:53:00.295514 20475 raft_consensus.cc:3037] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:00.296341 20344 raft_consensus.cc:2454] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc95658679904c3d8a879c794c6be327 in term 1.
I20260925 14:53:00.296342 20475 raft_consensus.cc:2454] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc95658679904c3d8a879c794c6be327 in term 1.
I20260925 14:53:00.296519 20149 leader_election.cc:302] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35e306cdd88f45e68452f62bdc655980, cc95658679904c3d8a879c794c6be327; no voters:
I20260925 14:53:00.296622 20540 raft_consensus.cc:2781] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:00.296757 20540 raft_consensus.cc:686] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [term 1 LEADER]: Becoming Leader. State: Replica: cc95658679904c3d8a879c794c6be327, State: Running, Role: LEADER
I20260925 14:53:00.296864 20540 consensus_queue.cc:237] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } }
I20260925 14:53:00.297662 20070 catalog_manager.cc:5495] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc95658679904c3d8a879c794c6be327 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc95658679904c3d8a879c794c6be327" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc95658679904c3d8a879c794c6be327" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40785 } health_report { overall_health: HEALTHY } } }
W20260925 14:53:00.341310 20522 tablet.cc:2369] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:53:00.342801 20260 tablet.cc:2369] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:53:00.345527 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.342860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.0.98.1:40785)
0925 14:53:00.342910 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:00.342912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:00.342922 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:00.342967 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:00.342967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:00.342998 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:00.343089 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:00.343235 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.343770 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.344654 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:00.344868 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:00.344964 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:00.345137 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:00.345256 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:00.345364 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:00.345386 (+ 22us) server_negotiation.cc:299] Negotiation successful
0925 14:53:00.345409 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:53:00.347297 20344 raft_consensus.cc:1261] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 1 FOLLOWER]: Refusing update from remote peer cc95658679904c3d8a879c794c6be327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:00.347306 20475 raft_consensus.cc:1261] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 1 FOLLOWER]: Refusing update from remote peer cc95658679904c3d8a879c794c6be327: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:00.347692 20545 consensus_queue.cc:1035] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35e306cdd88f45e68452f62bdc655980" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:00.347857 20540 consensus_queue.cc:1035] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fe5f253d13c40e08f16b2c46b8cd619" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:00.350512 20549 mvcc.cc:205] Tried to move back new op lower bound from 7333265327499722752 to 7333265327296454656. Current Snapshot: MvccSnapshot[applied={T|T < 7333265327499722752}]
I20260925 14:53:00.350744 20550 mvcc.cc:205] Tried to move back new op lower bound from 7333265327499722752 to 7333265327296454656. Current Snapshot: MvccSnapshot[applied={T|T < 7333265327499722752}]
I20260925 14:53:00.352790 20071 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:00.352906 20071 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:53:00.353710 20071 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c0cfd9da54644b88bc43853e17bfc04: Sending DeleteTablet for 3 replicas of tablet f07efbb9c17947ac8d151f1ea2bcf459
I20260925 14:53:00.354036 20324 tablet_service.cc:1514] Processing DeleteTablet for tablet f07efbb9c17947ac8d151f1ea2bcf459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:00 UTC) from {username='slave'} at 127.0.0.1:42166
I20260925 14:53:00.354149 20455 tablet_service.cc:1514] Processing DeleteTablet for tablet f07efbb9c17947ac8d151f1ea2bcf459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:00 UTC) from {username='slave'} at 127.0.0.1:36750
I20260925 14:53:00.354282 20558 tablet_replica.cc:330] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: stopping tablet replica
I20260925 14:53:00.354413 20193 tablet_service.cc:1514] Processing DeleteTablet for tablet f07efbb9c17947ac8d151f1ea2bcf459 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:53:00 UTC) from {username='slave'} at 127.0.0.1:45360
I20260925 14:53:00.354458 20558 raft_consensus.cc:2229] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:00.354588 20558 raft_consensus.cc:2258] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:00.354671 20560 tablet_replica.cc:330] T f07efbb9c17947ac8d151f1ea2bcf459 P cc95658679904c3d8a879c794c6be327: stopping tablet replica
I20260925 14:53:00.354766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20131
I20260925 14:53:00.355914 20558 ts_tablet_manager.cc:1905] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:00.356179 20559 tablet_replica.cc:330] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: stopping tablet replica
I20260925 14:53:00.356333 20559 raft_consensus.cc:2229] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:53:00.356457 20559 raft_consensus.cc:2258] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:53:00.356942 20559 ts_tablet_manager.cc:1905] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:53:00.357791 20558 ts_tablet_manager.cc:1918] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:00.357878 20558 log.cc:1196] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-1/wal/wals/f07efbb9c17947ac8d151f1ea2bcf459
I20260925 14:53:00.358223 20558 ts_tablet_manager.cc:1939] T f07efbb9c17947ac8d151f1ea2bcf459 P 2fe5f253d13c40e08f16b2c46b8cd619: Deleting consensus metadata
I20260925 14:53:00.358749 20559 ts_tablet_manager.cc:1918] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:53:00.358734 20057 catalog_manager.cc:4841] TS 2fe5f253d13c40e08f16b2c46b8cd619 (127.0.98.2:44445): tablet f07efbb9c17947ac8d151f1ea2bcf459 (table test-table [id=9ab406d5bdf04eb48d2eb786bd7cb91a]) successfully deleted
I20260925 14:53:00.358826 20559 log.cc:1196] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Deleting WAL directory at /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790347739045519-392-0/minicluster-data/ts-2/wal/wals/f07efbb9c17947ac8d151f1ea2bcf459
I20260925 14:53:00.359071 20559 ts_tablet_manager.cc:1939] T f07efbb9c17947ac8d151f1ea2bcf459 P 35e306cdd88f45e68452f62bdc655980: Deleting consensus metadata
I20260925 14:53:00.359504 20058 catalog_manager.cc:4841] TS 35e306cdd88f45e68452f62bdc655980 (127.0.98.3:36097): tablet f07efbb9c17947ac8d151f1ea2bcf459 (table test-table [id=9ab406d5bdf04eb48d2eb786bd7cb91a]) successfully deleted
W20260925 14:53:00.359854 20058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40785 (error 108)
I20260925 14:53:00.359982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20262
I20260925 14:53:00.361299 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:00.361137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40785 (local address 127.0.0.1:45374)
0925 14:53:00.361214 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:00.361260 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 14:53:00.361426 20058 catalog_manager.cc:4584] TS cc95658679904c3d8a879c794c6be327 (127.0.98.1:40785): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f07efbb9c17947ac8d151f1ea2bcf459: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40785: connect: Connection refused (error 111)
I20260925 14:53:00.365103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20393
I20260925 14:53:00.369524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20040
2026-09-25T14:53:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (845 ms)
[----------] 6 tests from EncryptionPolicyTest (8765 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:00 dist-test-slave-z2s5 krb5kdc[20564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:00 dist-test-slave-z2s5 krb5kdc[20564](info): set up 2 sockets
Sep 25 14:53:00 dist-test-slave-z2s5 krb5kdc[20564](info): commencing operation
krb5kdc: starting...
W20260925 14:53:01.410380 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:01 dist-test-slave-z2s5 krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:01.528837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:01.610186 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:01.610430 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:01.610455 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:01.611644 20577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:01.611690 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:01.611704 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:01.611717 20577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:01.611729 20577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:01.613199 20577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:01.613454 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:01.613703 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:01.618646 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:01.618703 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:01.618779 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:01.618986 20577 server_base.cc:1034] running on GCE node
I20260925 14:53:01.619295 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:01.619588 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:01.620743 20577 hybrid_clock.cc:648] HybridClock initialized: now 1790347981620724 us; error 32 us; skew 500 ppm
Sep 25 14:53:01 dist-test-slave-z2s5 krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:01.623420 20577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:01.623893 20577 webserver.cc:460] Webserver started at http://127.0.0.1:46443/ using document root <none> and password file <none>
I20260925 14:53:01.624116 20577 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:01.624161 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:01.624312 20577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:01.625365 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95"
format_stamp: "Formatted at 2026-09-25 14:53:01 on dist-test-slave-z2s5"
server_key: "53ff9492a04a0249ed6e8730050ab41c"
server_key_iv: "326a11fb8419eddc61bb0745e3ef7031"
server_key_version: "encryptionkey@0"
I20260925 14:53:01.625741 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95"
format_stamp: "Formatted at 2026-09-25 14:53:01 on dist-test-slave-z2s5"
server_key: "53ff9492a04a0249ed6e8730050ab41c"
server_key_iv: "326a11fb8419eddc61bb0745e3ef7031"
server_key_version: "encryptionkey@0"
I20260925 14:53:01.627259 20577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:01.628039 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:01.628252 20577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:01.628362 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95"
format_stamp: "Formatted at 2026-09-25 14:53:01 on dist-test-slave-z2s5"
server_key: "53ff9492a04a0249ed6e8730050ab41c"
server_key_iv: "326a11fb8419eddc61bb0745e3ef7031"
server_key_version: "encryptionkey@0"
I20260925 14:53:01.628455 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:01.651648 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:01.654162 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:01.654341 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:01.658499 20577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43819
I20260925 14:53:01.658536 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43819 every 8 connection(s)
I20260925 14:53:01.658905 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:01.659736 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:01.661936 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95: Bootstrap starting.
I20260925 14:53:01.662565 20645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:01.662803 20645 log.cc:824] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:01.663522 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95: No bootstrap required, opened a new log
I20260925 14:53:01.663970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20577
I20260925 14:53:01.664124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:01.664348 392 external_mini_cluster.cc:1428] Setting key 79d5beb88a602863c744ad1a2f209e36
I20260925 14:53:01.665669 20645 raft_consensus.cc:348] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43819 } }
I20260925 14:53:01.665848 20645 raft_consensus.cc:374] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:01.665885 20645 raft_consensus.cc:729] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d897dfa3a9b4407ab20ee90ae7ed0c95, State: Initialized, Role: FOLLOWER
I20260925 14:53:01.666042 20645 consensus_queue.cc:260] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43819 } }
I20260925 14:53:01.666106 20645 raft_consensus.cc:388] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:01.666127 20645 raft_consensus.cc:482] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:01.666180 20645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:01.667295 20645 raft_consensus.cc:504] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43819 } }
I20260925 14:53:01.667415 20645 leader_election.cc:302] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d897dfa3a9b4407ab20ee90ae7ed0c95; no voters:
I20260925 14:53:01.667563 20645 leader_election.cc:288] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:01.667644 20650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:01.667795 20645 sys_catalog.cc:564] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:01.667896 20650 raft_consensus.cc:686] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [term 1 LEADER]: Becoming Leader. State: Replica: d897dfa3a9b4407ab20ee90ae7ed0c95, State: Running, Role: LEADER
I20260925 14:53:01.667995 20650 consensus_queue.cc:237] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43819 } }
I20260925 14:53:01.668392 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d897dfa3a9b4407ab20ee90ae7ed0c95. Latest consensus state: current_term: 1 leader_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43819 } } }
I20260925 14:53:01.668476 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:01.669305 20651 sys_catalog.cc:455] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d897dfa3a9b4407ab20ee90ae7ed0c95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43819 } } }
I20260925 14:53:01.670013 20651 sys_catalog.cc:458] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95 [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:01.669548 20665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:01.670116 20665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:01.670197 20659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:01.670377 20659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:01.672063 20659 catalog_manager.cc:1347] Generated new cluster ID: bba04628f52647de977de9ec7b29602b
I20260925 14:53:01.672147 20659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:01 dist-test-slave-z2s5 krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347981, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:53:01.677564 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.664939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57867 (local address 127.0.0.1:43819)
0925 14:53:01.665140 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.665150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.665191 (+ 41us) server_negotiation.cc:407] Connection header received
0925 14:53:01.665624 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.665633 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.665695 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.665850 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.666776 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.668201 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.669547 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.670184 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.670304 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.673444 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:01.673448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:01.673449 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:01.673461 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:01.675653 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.676205 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.676208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.676208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.676348 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.676601 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.676601 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.676602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.676924 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:01.677100 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.677239 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.677349 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260925 14:53:01.678622 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:01.682049 20659 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:01.682833 20659 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:01.687955 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:01.680153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38305 (local address 127.0.0.1:43819)
0925 14:53:01.680194 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:01.680197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:01.680328 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:53:01.680372 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:01.680373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:01.680383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:01.680443 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:01.681117 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.681889 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.683022 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:01.683363 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:01.683436 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:01.684923 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:01.684925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:01.684925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:01.684933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:01.686640 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.687107 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.687108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.687109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.687171 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:01.687404 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:01.687405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:01.687406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:01.687537 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:01.687642 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:01.687790 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:53:01.687857 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:01.692930 20659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d897dfa3a9b4407ab20ee90ae7ed0c95: Generated new TSK 0
I20260925 14:53:01.693147 20659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:01.699590 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20577
W20260925 14:53:01.703961 20562 connection.cc:535] client connection to 127.0.0.1:43819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43819): Connection reset by peer (error 104)
2026-09-25T14:53:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:01 dist-test-slave-z2s5 krb5kdc[20676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:01 dist-test-slave-z2s5 krb5kdc[20676](info): set up 2 sockets
Sep 25 14:53:01 dist-test-slave-z2s5 krb5kdc[20676](info): commencing operation
krb5kdc: starting...
W20260925 14:53:02.774365 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:02 dist-test-slave-z2s5 krb5kdc[20676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:02.893673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33785
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:02.975703 20689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:02.975914 20689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:02.975936 20689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:02.977109 20689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:02.977152 20689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:02.977166 20689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:02.977180 20689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:02.977191 20689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:02.978827 20689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33785
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:02.979099 20689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:02.979333 20689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:02.984095 20695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:02.984165 20697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:02.984114 20694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:02.984436 20689 server_base.cc:1034] running on GCE node
I20260925 14:53:02.984683 20689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:02.984974 20689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:02.986122 20689 hybrid_clock.cc:648] HybridClock initialized: now 1790347982986109 us; error 30 us; skew 500 ppm
Sep 25 14:53:02 dist-test-slave-z2s5 krb5kdc[20676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:02.988929 20689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:02.989352 20689 webserver.cc:460] Webserver started at http://127.0.0.1:44539/ using document root <none> and password file <none>
I20260925 14:53:02.989562 20689 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:02.989606 20689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:02.989699 20689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:02.990869 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf"
format_stamp: "Formatted at 2026-09-25 14:53:02 on dist-test-slave-z2s5"
server_key: "039f0281d2f97773eacb60004cd55fcf"
server_key_iv: "0dce7dcea1763f4ae73d8f1fd43a5023"
server_key_version: "encryptionkey@0"
I20260925 14:53:02.991202 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf"
format_stamp: "Formatted at 2026-09-25 14:53:02 on dist-test-slave-z2s5"
server_key: "039f0281d2f97773eacb60004cd55fcf"
server_key_iv: "0dce7dcea1763f4ae73d8f1fd43a5023"
server_key_version: "encryptionkey@0"
I20260925 14:53:02.992599 20689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:02.993355 20704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:02.993533 20689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:02.993616 20689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf"
format_stamp: "Formatted at 2026-09-25 14:53:02 on dist-test-slave-z2s5"
server_key: "039f0281d2f97773eacb60004cd55fcf"
server_key_iv: "0dce7dcea1763f4ae73d8f1fd43a5023"
server_key_version: "encryptionkey@0"
I20260925 14:53:02.993672 20689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:03.012598 20689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:03.014611 20689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:03.014724 20689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:03.018678 20689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33785
I20260925 14:53:03.018769 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33785 every 8 connection(s)
I20260925 14:53:03.019115 20689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:03.020123 20757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:03.022706 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf: Bootstrap starting.
I20260925 14:53:03.023330 20757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:03.023641 20757 log.cc:824] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:03.024410 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf: No bootstrap required, opened a new log
I20260925 14:53:03.026171 20757 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33785 } }
I20260925 14:53:03.026351 20757 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:03.026432 20757 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3a22dd7ef2f42ebad5f512dc3f5cebf, State: Initialized, Role: FOLLOWER
I20260925 14:53:03.026643 20757 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33785 } }
I20260925 14:53:03.026765 20757 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:03.026818 20757 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:03.026876 20757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:03.027457 20757 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33785 } }
I20260925 14:53:03.027585 20757 leader_election.cc:302] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3a22dd7ef2f42ebad5f512dc3f5cebf; no voters:
I20260925 14:53:03.027812 20757 leader_election.cc:288] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:03.027858 20760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:03.028071 20760 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [term 1 LEADER]: Becoming Leader. State: Replica: d3a22dd7ef2f42ebad5f512dc3f5cebf, State: Running, Role: LEADER
I20260925 14:53:03.028095 20757 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:03.028162 20760 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33785 } }
I20260925 14:53:03.028592 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33785 } } }
I20260925 14:53:03.028597 20762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3a22dd7ef2f42ebad5f512dc3f5cebf. Latest consensus state: current_term: 1 leader_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3a22dd7ef2f42ebad5f512dc3f5cebf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33785 } } }
I20260925 14:53:03.028707 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:03.028790 20762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:03.028880 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20689
I20260925 14:53:03.029026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:03.029035 20764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:03.029271 392 external_mini_cluster.cc:1428] Setting key 29b528abf8d35d59c0e14a2a66ff75e5
I20260925 14:53:03.029565 20764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:03.031275 20764 catalog_manager.cc:1347] Generated new cluster ID: b938d7aff5404299acb41683bbb8ef1c
I20260925 14:53:03.031332 20764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:03.036290 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:03.029954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46813 (local address 127.0.0.1:33785)
0925 14:53:03.030640 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:03.030762 (+ 122us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:03.030793 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:53:03.030919 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:03.030928 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:03.030989 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:03.031142 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:03.031445 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.032795 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.034041 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.034684 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.034785 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:03.035587 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:03.035591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:03.035591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:03.035794 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:03.035831 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:03.035844 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:03.035921 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:03.036048 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:53:03.036074 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":86,"threads_started":1}
I20260925 14:53:03.037581 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:03.040808 20764 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:03.041462 20764 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:03.043105 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:03.039263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34173 (local address 127.0.0.1:33785)
0925 14:53:03.039305 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:03.039307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:03.039418 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:53:03.039482 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:03.039483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:03.039506 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:03.039575 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:03.039717 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.040546 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.041626 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:03.041986 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:03.042067 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:03.042840 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:03.042842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:03.042842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:03.042848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:03.042863 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:03.042868 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:03.042930 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:03.043011 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:53:03.043024 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:03.051528 20764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3a22dd7ef2f42ebad5f512dc3f5cebf: Generated new TSK 0
I20260925 14:53:03.051740 20764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:03.054688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20689
W20260925 14:53:03.058936 20674 connection.cc:535] client connection to 127.0.0.1:33785 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33785): Connection reset by peer (error 104)
2026-09-25T14:53:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:03 dist-test-slave-z2s5 krb5kdc[20788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:03 dist-test-slave-z2s5 krb5kdc[20788](info): set up 2 sockets
Sep 25 14:53:03 dist-test-slave-z2s5 krb5kdc[20788](info): commencing operation
krb5kdc: starting...
W20260925 14:53:04.118557 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:04 dist-test-slave-z2s5 krb5kdc[20788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:04.237826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:04.319548 20801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:04.319818 20801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:04.319856 20801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:04.321070 20801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:04.321151 20801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:04.321175 20801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:04.321205 20801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:04.321244 20801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:04.322784 20801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:04.323081 20801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:04.323326 20801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:04.328398 20807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.328398 20809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:04.328578 20806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:04.328815 20801 server_base.cc:1034] running on GCE node
I20260925 14:53:04.329063 20801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:04.329360 20801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:04.330468 20801 hybrid_clock.cc:648] HybridClock initialized: now 1790347984330439 us; error 37 us; skew 500 ppm
Sep 25 14:53:04 dist-test-slave-z2s5 krb5kdc[20788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:04.339924 20801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:04.340431 20801 webserver.cc:460] Webserver started at http://127.0.0.1:39319/ using document root <none> and password file <none>
I20260925 14:53:04.340662 20801 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:04.340708 20801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:04.340826 20801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:04.341917 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "4ce383c53e684a4a865b350ce2652864"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-z2s5"
server_key: "82e5ae3fc02bdca0f039e77215d340d8"
server_key_iv: "11eaaa27933b6c4097bdd3e02c4326a5"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.342278 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ce383c53e684a4a865b350ce2652864"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-z2s5"
server_key: "82e5ae3fc02bdca0f039e77215d340d8"
server_key_iv: "11eaaa27933b6c4097bdd3e02c4326a5"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.343689 20801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:04.344527 20816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:04.344729 20801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:04.344834 20801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "4ce383c53e684a4a865b350ce2652864"
format_stamp: "Formatted at 2026-09-25 14:53:04 on dist-test-slave-z2s5"
server_key: "82e5ae3fc02bdca0f039e77215d340d8"
server_key_iv: "11eaaa27933b6c4097bdd3e02c4326a5"
server_key_version: "encryptionkey@0"
I20260925 14:53:04.344910 20801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:04.348896 20801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:04.351421 20801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:04.351614 20801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:04.356238 20801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39533
I20260925 14:53:04.356269 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39533 every 8 connection(s)
I20260925 14:53:04.356613 20801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:04.357632 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:04.360044 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864: Bootstrap starting.
I20260925 14:53:04.360651 20869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:04.360953 20869 log.cc:824] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:04.361927 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864: No bootstrap required, opened a new log
I20260925 14:53:04.362890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20801
I20260925 14:53:04.363041 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:04.363231 392 external_mini_cluster.cc:1428] Setting key a8cf8415ea01f68ada13cd583ff96af2
I20260925 14:53:04.364197 20869 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce383c53e684a4a865b350ce2652864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39533 } }
I20260925 14:53:04.364374 20869 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:04.364413 20869 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ce383c53e684a4a865b350ce2652864, State: Initialized, Role: FOLLOWER
I20260925 14:53:04.364568 20869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce383c53e684a4a865b350ce2652864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39533 } }
I20260925 14:53:04.364669 20869 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:04.364722 20869 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:04.364773 20869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:04.365641 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.363785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56593 (local address 127.0.0.1:39533)
0925 14:53:04.364002 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.364014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.364058 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:53:04.364179 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.364187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.364239 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.364392 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.365135 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.365140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.365140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:04.365144 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.365206 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:04.365225 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.365305 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.365374 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.365425 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260925 14:53:04.365942 20869 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce383c53e684a4a865b350ce2652864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39533 } }
I20260925 14:53:04.366063 20869 leader_election.cc:302] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ce383c53e684a4a865b350ce2652864; no voters:
I20260925 14:53:04.366217 20869 leader_election.cc:288] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:04.366245 20874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:04.366356 20874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [term 1 LEADER]: Becoming Leader. State: Replica: 4ce383c53e684a4a865b350ce2652864, State: Running, Role: LEADER
I20260925 14:53:04.366498 20869 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:04.366590 20874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce383c53e684a4a865b350ce2652864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39533 } }
I20260925 14:53:04.366762 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:04.367873 20876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ce383c53e684a4a865b350ce2652864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce383c53e684a4a865b350ce2652864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39533 } } }
I20260925 14:53:04.367967 20876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:04.368360 20888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 14:53:04.368316 20875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ce383c53e684a4a865b350ce2652864. Latest consensus state: current_term: 1 leader_uuid: "4ce383c53e684a4a865b350ce2652864" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ce383c53e684a4a865b350ce2652864" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39533 } } }
W20260925 14:53:04.368481 20888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:04.368554 20875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:04.368880 20893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:04.369055 20893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:04.369861 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:04.368637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48081 (local address 127.0.0.1:39533)
0925 14:53:04.368676 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:04.368678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:04.368827 (+ 149us) server_negotiation.cc:407] Connection header received
0925 14:53:04.368908 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:04.368909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:04.368915 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:04.368963 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:04.369621 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:04.369622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:04.369623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:04.369627 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:04.369641 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:04.369645 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:04.369711 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:04.369764 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:53:04.369777 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:04.370978 20893 catalog_manager.cc:1347] Generated new cluster ID: 3fd88a94198e4c809770c39aeac11953
I20260925 14:53:04.371044 20893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:04.384400 20893 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:04.384600 20893 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:04.397171 20893 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ce383c53e684a4a865b350ce2652864: Generated new TSK 0
I20260925 14:53:04.397378 20893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:04.399379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20801
2026-09-25T14:53:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:04 dist-test-slave-z2s5 krb5kdc[20900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:04 dist-test-slave-z2s5 krb5kdc[20900](info): set up 2 sockets
Sep 25 14:53:04 dist-test-slave-z2s5 krb5kdc[20900](info): commencing operation
krb5kdc: starting...
W20260925 14:53:05.430557 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:05 dist-test-slave-z2s5 krb5kdc[20900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:05.548826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:05.631215 20913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:05.631446 20913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:05.631469 20913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:05.632669 20913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:05.632722 20913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:05.632743 20913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:05.632797 20913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:05.632815 20913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:05.634508 20913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:05.634779 20913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:05.635015 20913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:05.640007 20921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.640017 20918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:05.640089 20919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:05.640327 20913 server_base.cc:1034] running on GCE node
I20260925 14:53:05.640539 20913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:05.640779 20913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:05.641938 20913 hybrid_clock.cc:648] HybridClock initialized: now 1790347985641926 us; error 32 us; skew 500 ppm
Sep 25 14:53:05 dist-test-slave-z2s5 krb5kdc[20900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:05.644381 20913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:05.644774 20913 webserver.cc:460] Webserver started at http://127.0.0.1:37603/ using document root <none> and password file <none>
I20260925 14:53:05.644961 20913 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:05.645001 20913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:05.645092 20913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:05.646126 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "2c497c2540c84429989af8ab9794e893"
format_stamp: "Formatted at 2026-09-25 14:53:05 on dist-test-slave-z2s5"
server_key: "695896789427a213c7ace84e008962fe"
server_key_iv: "0818bdf35e71611734eef24f1a49f572"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.646520 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c497c2540c84429989af8ab9794e893"
format_stamp: "Formatted at 2026-09-25 14:53:05 on dist-test-slave-z2s5"
server_key: "695896789427a213c7ace84e008962fe"
server_key_iv: "0818bdf35e71611734eef24f1a49f572"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.647883 20913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.648644 20928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:05.648878 20913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:05.648999 20913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "2c497c2540c84429989af8ab9794e893"
format_stamp: "Formatted at 2026-09-25 14:53:05 on dist-test-slave-z2s5"
server_key: "695896789427a213c7ace84e008962fe"
server_key_iv: "0818bdf35e71611734eef24f1a49f572"
server_key_version: "encryptionkey@0"
I20260925 14:53:05.649082 20913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:05.667816 20913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:05.669764 20913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:05.669883 20913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:05.673813 20913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35533
I20260925 14:53:05.673851 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35533 every 8 connection(s)
I20260925 14:53:05.674208 20913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:05.675136 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:05.677346 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893: Bootstrap starting.
I20260925 14:53:05.677927 20981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:05.678208 20981 log.cc:824] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:05.678999 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893: No bootstrap required, opened a new log
I20260925 14:53:05.680547 20981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c497c2540c84429989af8ab9794e893" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35533 } }
I20260925 14:53:05.680686 20981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:05.680708 20981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c497c2540c84429989af8ab9794e893, State: Initialized, Role: FOLLOWER
I20260925 14:53:05.680874 20981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c497c2540c84429989af8ab9794e893" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35533 } }
I20260925 14:53:05.680958 20981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:05.680981 20981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:05.681051 20981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:05.681562 20981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c497c2540c84429989af8ab9794e893" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35533 } }
I20260925 14:53:05.681646 20981 leader_election.cc:302] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c497c2540c84429989af8ab9794e893; no voters:
I20260925 14:53:05.681830 20981 leader_election.cc:288] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:05.681879 20984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:05.682080 20984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [term 1 LEADER]: Becoming Leader. State: Replica: 2c497c2540c84429989af8ab9794e893, State: Running, Role: LEADER
I20260925 14:53:05.682120 20981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:05.682178 20984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c497c2540c84429989af8ab9794e893" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35533 } }
I20260925 14:53:05.682583 20986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c497c2540c84429989af8ab9794e893. Latest consensus state: current_term: 1 leader_uuid: "2c497c2540c84429989af8ab9794e893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c497c2540c84429989af8ab9794e893" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35533 } } }
I20260925 14:53:05.682669 20986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:05.682813 20985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c497c2540c84429989af8ab9794e893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c497c2540c84429989af8ab9794e893" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35533 } } }
I20260925 14:53:05.682919 20985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:05.683207 20993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:05.683717 20993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:05.683831 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 20913
I20260925 14:53:05.683944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:05.684126 392 external_mini_cluster.cc:1428] Setting key 4372bc52be0d8839ed86c2642aa348d4
I20260925 14:53:05.685437 20993 catalog_manager.cc:1347] Generated new cluster ID: 9f36a110505e42dea7ab6a40fd59d494
I20260925 14:53:05.685491 20993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:05 dist-test-slave-z2s5 krb5kdc[20900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347985, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:53:05.695736 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.684565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41247 (local address 127.0.0.1:35533)
0925 14:53:05.684765 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.684774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.684812 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:53:05.684941 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.684950 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.685013 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.685170 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:05.685960 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.687451 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.688493 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.689121 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.689240 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.692027 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:05.692032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:05.692032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:05.692045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:05.694024 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.694490 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.694493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.694494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.694599 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.694818 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.694819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.694819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.695111 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:05.695245 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.695377 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.695508 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
I20260925 14:53:05.696784 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:05.697715 20993 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:05.698519 20993 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:05.702162 20993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c497c2540c84429989af8ab9794e893: Generated new TSK 0
I20260925 14:53:05.702364 20993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:05.705601 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:05.698232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45775 (local address 127.0.0.1:35533)
0925 14:53:05.698267 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:05.698269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:05.698445 (+ 176us) server_negotiation.cc:407] Connection header received
0925 14:53:05.698476 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:05.698476 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:05.698513 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:05.698576 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:05.699223 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.699973 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.701083 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:05.701421 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:05.701485 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:05.702978 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:05.702979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:05.702979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:05.702985 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:05.704444 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.704882 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.704883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.704883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.704941 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:05.705139 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:05.705140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:05.705140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:05.705236 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:05.705314 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:05.705480 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:53:05.705541 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:05.707517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 20913
2026-09-25T14:53:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:05 dist-test-slave-z2s5 krb5kdc[21012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:05 dist-test-slave-z2s5 krb5kdc[21012](info): set up 2 sockets
Sep 25 14:53:05 dist-test-slave-z2s5 krb5kdc[21012](info): commencing operation
krb5kdc: starting...
W20260925 14:53:06.766525 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:06 dist-test-slave-z2s5 krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:06.893189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46141
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:06.973994 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:06.974210 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:06.974236 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:06.975373 21025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:06.975412 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:06.975427 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:06.975440 21025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:06.975452 21025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:06.976869 21025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:06.977145 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:06.977404 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:06.982450 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:06.982443 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:06.982443 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:06.982663 21025 server_base.cc:1034] running on GCE node
I20260925 14:53:06.983031 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:06.983275 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:06.984444 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790347986984430 us; error 28 us; skew 500 ppm
Sep 25 14:53:06 dist-test-slave-z2s5 krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:06.987357 21025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:06.987887 21025 webserver.cc:460] Webserver started at http://127.0.0.1:39213/ using document root <none> and password file <none>
I20260925 14:53:06.988159 21025 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:06.988209 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:06.988396 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:06.989580 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "d9c0624526d449179ec612eefbbd7ec1"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-z2s5"
server_key: "f0c2a0cab1540c6103e783b5afb7f8ab"
server_key_iv: "0e6d7cf7c6cd404dbef0903029c29116"
server_key_version: "encryptionkey@0"
I20260925 14:53:06.989967 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "d9c0624526d449179ec612eefbbd7ec1"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-z2s5"
server_key: "f0c2a0cab1540c6103e783b5afb7f8ab"
server_key_iv: "0e6d7cf7c6cd404dbef0903029c29116"
server_key_version: "encryptionkey@0"
I20260925 14:53:06.991499 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:06.992259 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:06.992448 21025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:06.992544 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "d9c0624526d449179ec612eefbbd7ec1"
format_stamp: "Formatted at 2026-09-25 14:53:06 on dist-test-slave-z2s5"
server_key: "f0c2a0cab1540c6103e783b5afb7f8ab"
server_key_iv: "0e6d7cf7c6cd404dbef0903029c29116"
server_key_version: "encryptionkey@0"
I20260925 14:53:06.992621 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:07.008280 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:07.010429 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:07.010598 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:07.014524 21025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46141
I20260925 14:53:07.014580 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46141 every 8 connection(s)
I20260925 14:53:07.014935 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:07.015782 21093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:07.017971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21025
I20260925 14:53:07.018108 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:07.018124 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1: Bootstrap starting.
I20260925 14:53:07.018283 392 external_mini_cluster.cc:1428] Setting key dae88ae09b7e264b29cda99f859dd281
I20260925 14:53:07.018870 21093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:07.019184 21093 log.cc:824] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:07.019919 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1: No bootstrap required, opened a new log
I20260925 14:53:07.021982 21093 raft_consensus.cc:348] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c0624526d449179ec612eefbbd7ec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260925 14:53:07.022161 21093 raft_consensus.cc:374] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:07.022254 21093 raft_consensus.cc:729] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c0624526d449179ec612eefbbd7ec1, State: Initialized, Role: FOLLOWER
I20260925 14:53:07.022439 21093 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c0624526d449179ec612eefbbd7ec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260925 14:53:07.022523 21093 raft_consensus.cc:388] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:07.022547 21093 raft_consensus.cc:482] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:07.022609 21093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:07.023326 21093 raft_consensus.cc:504] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c0624526d449179ec612eefbbd7ec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260925 14:53:07.023435 21093 leader_election.cc:302] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9c0624526d449179ec612eefbbd7ec1; no voters:
I20260925 14:53:07.023603 21093 leader_election.cc:288] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:07.023633 21098 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:07.023808 21098 raft_consensus.cc:686] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [term 1 LEADER]: Becoming Leader. State: Replica: d9c0624526d449179ec612eefbbd7ec1, State: Running, Role: LEADER
I20260925 14:53:07.023854 21093 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:07.023960 21098 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c0624526d449179ec612eefbbd7ec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } }
I20260925 14:53:07.024336 21100 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9c0624526d449179ec612eefbbd7ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c0624526d449179ec612eefbbd7ec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } } }
I20260925 14:53:07.024326 21098 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9c0624526d449179ec612eefbbd7ec1. Latest consensus state: current_term: 1 leader_uuid: "d9c0624526d449179ec612eefbbd7ec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9c0624526d449179ec612eefbbd7ec1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46141 } } }
I20260925 14:53:07.024441 21100 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:07.024444 21098 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:07.024344 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.018845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40249 (local address 127.0.0.1:46141)
0925 14:53:07.019091 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:07.019099 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:07.019124 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:53:07.019614 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:07.019621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:07.019672 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:07.019790 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:07.020054 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.020952 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.022121 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.022775 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.022860 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.023509 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:07.023513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:07.023513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:07.023737 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:07.023776 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:07.023791 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:07.024045 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:07.024080 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:53:07.024155 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":113,"threads_started":1}
I20260925 14:53:07.025405 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:07.025377 21105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:07.025868 21105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:07.027333 21105 catalog_manager.cc:1347] Generated new cluster ID: d92ac94b6a0b416fb14f6d0c8bbf2b8e
I20260925 14:53:07.027388 21105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:07.033346 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:07.026894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49129 (local address 127.0.0.1:46141)
0925 14:53:07.026936 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:07.026938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:07.030610 (+ 3672us) server_negotiation.cc:407] Connection header received
0925 14:53:07.030657 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:07.030658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:07.030668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:07.030724 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:07.030864 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.031357 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.032084 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:07.032293 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:07.032377 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:07.033053 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:07.033054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:07.033054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:07.033059 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:07.033073 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:07.033077 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:07.033128 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:07.033190 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 14:53:07.033247 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:53:07.043298 21105 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:07.044046 21105 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:07.063593 21105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d9c0624526d449179ec612eefbbd7ec1: Generated new TSK 0
I20260925 14:53:07.063850 21105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:07.068194 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21025
2026-09-25T14:53:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:07 dist-test-slave-z2s5 krb5kdc[21124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:07 dist-test-slave-z2s5 krb5kdc[21124](info): set up 2 sockets
Sep 25 14:53:07 dist-test-slave-z2s5 krb5kdc[21124](info): commencing operation
krb5kdc: starting...
W20260925 14:53:08.122625 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:08 dist-test-slave-z2s5 krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:08.241884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:08.321628 21137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:08.321872 21137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:08.321919 21137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:08.323073 21137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:08.323144 21137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:08.323167 21137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:08.323189 21137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:08.323217 21137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:08.324649 21137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:08.324919 21137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:08.325143 21137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:08.329953 21145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:08.329934 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:08.330219 21137 server_base.cc:1034] running on GCE node
W20260925 14:53:08.329941 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:08.330613 21137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:08.330904 21137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:08.332077 21137 hybrid_clock.cc:648] HybridClock initialized: now 1790347988332032 us; error 92 us; skew 500 ppm
Sep 25 14:53:08 dist-test-slave-z2s5 krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:08.339728 21137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:08.340266 21137 webserver.cc:460] Webserver started at http://127.0.0.1:44191/ using document root <none> and password file <none>
I20260925 14:53:08.340500 21137 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:08.340570 21137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:08.340720 21137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:08.341787 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "b0e44f6df2a740a694ae3ac23c103d2c"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-z2s5"
server_key: "f322366ecc5ae30894e625de87ec390a"
server_key_iv: "a1b413492e4fd41231cbc455296cdb34"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.342162 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0e44f6df2a740a694ae3ac23c103d2c"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-z2s5"
server_key: "f322366ecc5ae30894e625de87ec390a"
server_key_iv: "a1b413492e4fd41231cbc455296cdb34"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.343523 21137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:08.344290 21152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:08.344476 21137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:08.344574 21137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "b0e44f6df2a740a694ae3ac23c103d2c"
format_stamp: "Formatted at 2026-09-25 14:53:08 on dist-test-slave-z2s5"
server_key: "f322366ecc5ae30894e625de87ec390a"
server_key_iv: "a1b413492e4fd41231cbc455296cdb34"
server_key_version: "encryptionkey@0"
I20260925 14:53:08.344661 21137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:08.347981 21137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:08.349807 21137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:08.349972 21137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:08.353837 21137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45943
I20260925 14:53:08.353894 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45943 every 8 connection(s)
I20260925 14:53:08.354218 21137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:08.355211 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:08.356464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21137
I20260925 14:53:08.356593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:08.356781 392 external_mini_cluster.cc:1428] Setting key d9081c44e670c922becc0ff4adc61320
I20260925 14:53:08.357837 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c: Bootstrap starting.
I20260925 14:53:08.358556 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:08.358851 21205 log.cc:824] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:08.359539 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.357306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49519 (local address 127.0.0.1:45943)
0925 14:53:08.357502 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.357512 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.357542 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:53:08.358081 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.358090 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.358124 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.358291 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.359045 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.359048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.359048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:08.359052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.359087 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:08.359098 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.359172 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.359262 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.359325 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260925 14:53:08.359627 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c: No bootstrap required, opened a new log
I20260925 14:53:08.360548 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:08.362361 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260925 14:53:08.362581 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:08.362619 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e44f6df2a740a694ae3ac23c103d2c, State: Initialized, Role: FOLLOWER
I20260925 14:53:08.362741 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260925 14:53:08.362849 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:08.362895 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:08.362946 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:08.363479 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:08.362227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56489 (local address 127.0.0.1:45943)
0925 14:53:08.362263 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:08.362265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:08.362374 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:53:08.362437 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:08.362438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:08.362442 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:08.362521 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:08.363183 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:08.363184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:08.363184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:08.363189 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:08.363265 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:08.363271 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:08.363343 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:08.363403 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 14:53:08.363417 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:08.363780 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260925 14:53:08.363917 21205 leader_election.cc:302] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e44f6df2a740a694ae3ac23c103d2c; no voters:
I20260925 14:53:08.364107 21205 leader_election.cc:288] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:08.364213 21215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:08.364378 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:08.364441 21215 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [term 1 LEADER]: Becoming Leader. State: Replica: b0e44f6df2a740a694ae3ac23c103d2c, State: Running, Role: LEADER
I20260925 14:53:08.364666 21215 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } }
I20260925 14:53:08.365609 21217 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } } }
I20260925 14:53:08.365777 21217 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:08.365948 21216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e44f6df2a740a694ae3ac23c103d2c. Latest consensus state: current_term: 1 leader_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e44f6df2a740a694ae3ac23c103d2c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45943 } } }
I20260925 14:53:08.366168 21230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:08.366195 21216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:08.367038 21230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:08.374694 21230 catalog_manager.cc:1347] Generated new cluster ID: d922354cf4d64364be152cea96e7efe1
I20260925 14:53:08.374774 21230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:08.390722 21230 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:08.391026 21230 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:08.401820 21230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e44f6df2a740a694ae3ac23c103d2c: Generated new TSK 0
I20260925 14:53:08.402036 21230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:08.405879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21137
2026-09-25T14:53:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:08 dist-test-slave-z2s5 krb5kdc[21236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:08 dist-test-slave-z2s5 krb5kdc[21236](info): set up 2 sockets
Sep 25 14:53:08 dist-test-slave-z2s5 krb5kdc[21236](info): commencing operation
krb5kdc: starting...
W20260925 14:53:09.438627 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:09 dist-test-slave-z2s5 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:09.557816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:09.638993 21249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:09.639253 21249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:09.639293 21249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:09.640543 21249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:09.640618 21249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:09.640643 21249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:09.640664 21249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:09.640691 21249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:09.642241 21249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:09.642547 21249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:09.642786 21249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:09.647801 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:09.647825 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:09.647841 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:09.648012 21249 server_base.cc:1034] running on GCE node
I20260925 14:53:09.648455 21249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:09.648777 21249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:09.649978 21249 hybrid_clock.cc:648] HybridClock initialized: now 1790347989649993 us; error 62 us; skew 500 ppm
Sep 25 14:53:09 dist-test-slave-z2s5 krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:09.659466 21249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:09.659998 21249 webserver.cc:460] Webserver started at http://127.0.0.1:36549/ using document root <none> and password file <none>
I20260925 14:53:09.660241 21249 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:09.660287 21249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:09.660444 21249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:09.661540 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f"
format_stamp: "Formatted at 2026-09-25 14:53:09 on dist-test-slave-z2s5"
server_key: "a2b6c5167dbf15394fb6afcb8ff7a51c"
server_key_iv: "d30d976468d8d83b22409cab12dc0882"
server_key_version: "encryptionkey@0"
I20260925 14:53:09.661926 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f"
format_stamp: "Formatted at 2026-09-25 14:53:09 on dist-test-slave-z2s5"
server_key: "a2b6c5167dbf15394fb6afcb8ff7a51c"
server_key_iv: "d30d976468d8d83b22409cab12dc0882"
server_key_version: "encryptionkey@0"
I20260925 14:53:09.663414 21249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:09.664235 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:09.664458 21249 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:09.664551 21249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f"
format_stamp: "Formatted at 2026-09-25 14:53:09 on dist-test-slave-z2s5"
server_key: "a2b6c5167dbf15394fb6afcb8ff7a51c"
server_key_iv: "d30d976468d8d83b22409cab12dc0882"
server_key_version: "encryptionkey@0"
I20260925 14:53:09.664628 21249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:09.683290 21249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:09.685505 21249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:09.685623 21249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:09.690001 21249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39759
I20260925 14:53:09.690042 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39759 every 8 connection(s)
I20260925 14:53:09.690500 21249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:09.691349 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:09.692721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21249
I20260925 14:53:09.692840 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:09.693049 392 external_mini_cluster.cc:1428] Setting key 889cef3c57953f13659c85e1a5dd8f36
I20260925 14:53:09.693970 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f: Bootstrap starting.
I20260925 14:53:09.694620 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:09.694900 21317 log.cc:824] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:09.695573 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f: No bootstrap required, opened a new log
I20260925 14:53:09.697324 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260925 14:53:09.697502 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:09.697566 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b838c1e8ec64f60a9e76d1bbcdfb42f, State: Initialized, Role: FOLLOWER
I20260925 14:53:09.697729 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260925 14:53:09.697829 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:09.697872 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:09.697921 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:09.698506 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260925 14:53:09.698614 21317 leader_election.cc:302] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b838c1e8ec64f60a9e76d1bbcdfb42f; no voters:
I20260925 14:53:09.698807 21317 leader_election.cc:288] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:09.698860 21322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:09.699012 21322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [term 1 LEADER]: Becoming Leader. State: Replica: 1b838c1e8ec64f60a9e76d1bbcdfb42f, State: Running, Role: LEADER
I20260925 14:53:09.699070 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:09.699191 21322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } }
I20260925 14:53:09.699602 21323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } } }
I20260925 14:53:09.699612 21324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b838c1e8ec64f60a9e76d1bbcdfb42f. Latest consensus state: current_term: 1 leader_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b838c1e8ec64f60a9e76d1bbcdfb42f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39759 } } }
I20260925 14:53:09.699751 21323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [sys.catalog]: This master's current role is: LEADER
Sep 25 14:53:09 dist-test-slave-z2s5 krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347989, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:53:09.699762 21324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:09.700103 21331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:09.700649 21331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:09.703747 21331 catalog_manager.cc:1347] Generated new cluster ID: 08671881bd494fd2987f19515b41bde7
I20260925 14:53:09.703814 21331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:09.704268 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:09.693646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:47435 (local address 127.0.0.1:39759)
0925 14:53:09.693826 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:09.693844 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:09.693876 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:53:09.694321 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:09.694328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:09.694381 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:09.694588 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:09.695272 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:09.696222 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:09.697000 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:09.697406 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:09.697526 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:09.700450 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:09.700454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:09.700455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:09.700467 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:09.702354 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:09.702881 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:09.702885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:09.702886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:09.703002 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:09.703282 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:09.703283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:09.703284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:09.703574 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:09.703702 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:09.703926 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 14:53:09.704053 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260925 14:53:09.705569 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:09.709270 21331 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:09.709968 21331 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:09.714562 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:09.707127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:54131 (local address 127.0.0.1:39759)
0925 14:53:09.707170 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:09.707172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:09.707184 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:53:09.707254 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:09.707255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:09.707270 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:09.707324 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:09.707942 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:09.708708 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:09.709879 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:09.710204 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:09.710280 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:09.711725 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:09.711726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:09.711727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:09.711734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:09.713352 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:09.713752 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:09.713753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:09.713753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:09.713813 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:09.714040 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:09.714040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:09.714041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:09.714168 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:09.714273 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:09.714405 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:53:09.714469 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":22}
I20260925 14:53:09.715417 21331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b838c1e8ec64f60a9e76d1bbcdfb42f: Generated new TSK 0
I20260925 14:53:09.715600 21331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:09.721632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21249
W20260925 14:53:09.725879 21234 connection.cc:535] client connection to 127.0.0.1:39759 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39759): Connection reset by peer (error 104)
2026-09-25T14:53:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:09 dist-test-slave-z2s5 krb5kdc[21348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:09 dist-test-slave-z2s5 krb5kdc[21348](info): set up 2 sockets
Sep 25 14:53:09 dist-test-slave-z2s5 krb5kdc[21348](info): commencing operation
krb5kdc: starting...
W20260925 14:53:10.750643 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:10 dist-test-slave-z2s5 krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:10.880813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:10.962780 21361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:10.963016 21361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:10.963040 21361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:10.964172 21361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:10.964207 21361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:10.964222 21361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:10.964236 21361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:10.964248 21361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:10.965687 21361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:10.965898 21361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:10.966068 21361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:10.971115 21367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.971107 21369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:10.971107 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:10.971441 21361 server_base.cc:1034] running on GCE node
I20260925 14:53:10.971720 21361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:10.972028 21361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:10.973228 21361 hybrid_clock.cc:648] HybridClock initialized: now 1790347990973228 us; error 44 us; skew 500 ppm
Sep 25 14:53:10 dist-test-slave-z2s5 krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:10.976006 21361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:10.976399 21361 webserver.cc:460] Webserver started at http://127.0.0.1:44833/ using document root <none> and password file <none>
I20260925 14:53:10.976578 21361 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:10.976617 21361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:10.976703 21361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:10.977715 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "16b61559aed848c4b48e826ccfb9d3c1"
format_stamp: "Formatted at 2026-09-25 14:53:10 on dist-test-slave-z2s5"
server_key: "aaa85de9706f406e4e1dc803148b0197"
server_key_iv: "c90b4adf2bb8de1448afc6f53aab8a60"
server_key_version: "encryptionkey@0"
I20260925 14:53:10.978024 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "16b61559aed848c4b48e826ccfb9d3c1"
format_stamp: "Formatted at 2026-09-25 14:53:10 on dist-test-slave-z2s5"
server_key: "aaa85de9706f406e4e1dc803148b0197"
server_key_iv: "c90b4adf2bb8de1448afc6f53aab8a60"
server_key_version: "encryptionkey@0"
I20260925 14:53:10.979417 21361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:10.980329 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:10.980525 21361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:10.980618 21361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "16b61559aed848c4b48e826ccfb9d3c1"
format_stamp: "Formatted at 2026-09-25 14:53:10 on dist-test-slave-z2s5"
server_key: "aaa85de9706f406e4e1dc803148b0197"
server_key_iv: "c90b4adf2bb8de1448afc6f53aab8a60"
server_key_version: "encryptionkey@0"
I20260925 14:53:10.980674 21361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:11.000541 21361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:11.002910 21361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:11.003036 21361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:11.007411 21361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43367
I20260925 14:53:11.007772 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43367 every 8 connection(s)
I20260925 14:53:11.007823 21361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:11.008479 21429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:11.010852 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1: Bootstrap starting.
I20260925 14:53:11.011463 21429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:11.011751 21429 log.cc:824] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:11.012472 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1: No bootstrap required, opened a new log
I20260925 14:53:11.014155 21429 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b61559aed848c4b48e826ccfb9d3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260925 14:53:11.014328 21429 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:11.014379 21429 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b61559aed848c4b48e826ccfb9d3c1, State: Initialized, Role: FOLLOWER
I20260925 14:53:11.014582 21429 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b61559aed848c4b48e826ccfb9d3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260925 14:53:11.014667 21429 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:11.014711 21429 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:11.014765 21429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:11.015327 21429 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b61559aed848c4b48e826ccfb9d3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260925 14:53:11.015450 21429 leader_election.cc:302] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b61559aed848c4b48e826ccfb9d3c1; no voters:
I20260925 14:53:11.015654 21429 leader_election.cc:288] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:11.015806 21432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:11.016006 21429 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:11.016031 21432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [term 1 LEADER]: Becoming Leader. State: Replica: 16b61559aed848c4b48e826ccfb9d3c1, State: Running, Role: LEADER
I20260925 14:53:11.016247 21432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b61559aed848c4b48e826ccfb9d3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } }
I20260925 14:53:11.016304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21361
I20260925 14:53:11.016415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:11.016650 392 external_mini_cluster.cc:1428] Setting key 808277c35a456a446437e2293ea12bbd
I20260925 14:53:11.016952 21434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b61559aed848c4b48e826ccfb9d3c1. Latest consensus state: current_term: 1 leader_uuid: "16b61559aed848c4b48e826ccfb9d3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b61559aed848c4b48e826ccfb9d3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } } }
I20260925 14:53:11.017052 21434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:11.017112 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b61559aed848c4b48e826ccfb9d3c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b61559aed848c4b48e826ccfb9d3c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43367 } } }
I20260925 14:53:11.017169 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:11.017295 21443 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:11.017851 21443 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:11.019842 21443 catalog_manager.cc:1347] Generated new cluster ID: 24906b85695a4d4abf08bd530f52f52a
I20260925 14:53:11.019898 21443 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:11.022926 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.017332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:60917 (local address 127.0.0.1:43367)
0925 14:53:11.017551 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:11.017562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:11.017898 (+ 336us) server_negotiation.cc:407] Connection header received
0925 14:53:11.018059 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:11.018067 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:11.018135 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:11.018256 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:11.018669 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.020037 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.020861 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.021289 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.021405 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.022150 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:11.022154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:11.022155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:11.022438 (+ 283us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:11.022476 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:11.022490 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:11.022596 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:11.022686 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:53:11.022716 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260925 14:53:11.023828 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:11.028486 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:11.025215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:50123 (local address 127.0.0.1:43367)
0925 14:53:11.025250 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:11.025252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:11.025465 (+ 213us) server_negotiation.cc:407] Connection header received
0925 14:53:11.025537 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:11.025538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:11.025548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:11.025590 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:11.025740 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.026596 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.027320 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:11.027549 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:11.027634 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:11.028260 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:11.028262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:11.028262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:11.028266 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:11.028279 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:11.028283 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:11.028337 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:11.028393 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:53:11.028407 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:53:11.045457 21443 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:11.046059 21443 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:11.051203 21443 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b61559aed848c4b48e826ccfb9d3c1: Generated new TSK 0
I20260925 14:53:11.051437 21443 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:11.054288 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21361
W20260925 14:53:11.058444 21346 connection.cc:535] client connection to 127.0.0.1:43367 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43367): Connection reset by peer (error 104)
2026-09-25T14:53:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:11 dist-test-slave-z2s5 krb5kdc[21460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:11 dist-test-slave-z2s5 krb5kdc[21460](info): set up 2 sockets
Sep 25 14:53:11 dist-test-slave-z2s5 krb5kdc[21460](info): commencing operation
krb5kdc: starting...
W20260925 14:53:12.090703 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:12 dist-test-slave-z2s5 krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:12.213388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:12.300420 21473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:12.300648 21473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:12.300673 21473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:12.301872 21473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:12.301918 21473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:12.301932 21473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:12.301985 21473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:12.302003 21473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:12.303577 21473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:12.303839 21473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:12.304088 21473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:12.309403 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:12.309419 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:12.309440 21481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:12.309659 21473 server_base.cc:1034] running on GCE node
I20260925 14:53:12.309896 21473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:12.310150 21473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:12.311295 21473 hybrid_clock.cc:648] HybridClock initialized: now 1790347992311263 us; error 49 us; skew 500 ppm
Sep 25 14:53:12 dist-test-slave-z2s5 krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:12.313849 21473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:12.314354 21473 webserver.cc:460] Webserver started at http://127.0.0.1:42015/ using document root <none> and password file <none>
I20260925 14:53:12.314658 21473 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:12.314728 21473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:12.314865 21473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:12.315955 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "b02029c37bb54988a9aae9a3b83f6520"
format_stamp: "Formatted at 2026-09-25 14:53:12 on dist-test-slave-z2s5"
server_key: "d8b68b816d751a43047c557124e5cf50"
server_key_iv: "c65aea20634800b5824f2bf91eda7547"
server_key_version: "encryptionkey@0"
I20260925 14:53:12.316341 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "b02029c37bb54988a9aae9a3b83f6520"
format_stamp: "Formatted at 2026-09-25 14:53:12 on dist-test-slave-z2s5"
server_key: "d8b68b816d751a43047c557124e5cf50"
server_key_iv: "c65aea20634800b5824f2bf91eda7547"
server_key_version: "encryptionkey@0"
I20260925 14:53:12.317728 21473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:12.318506 21488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:12.318687 21473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:12.318792 21473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "b02029c37bb54988a9aae9a3b83f6520"
format_stamp: "Formatted at 2026-09-25 14:53:12 on dist-test-slave-z2s5"
server_key: "d8b68b816d751a43047c557124e5cf50"
server_key_iv: "c65aea20634800b5824f2bf91eda7547"
server_key_version: "encryptionkey@0"
I20260925 14:53:12.318879 21473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:12.322189 21473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:12.324146 21473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:12.324317 21473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:12.328733 21473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39795
I20260925 14:53:12.328768 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39795 every 8 connection(s)
I20260925 14:53:12.329121 21473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:12.330106 21541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:12.332499 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520: Bootstrap starting.
I20260925 14:53:12.333106 21541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:12.333397 21541 log.cc:824] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:12.334121 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520: No bootstrap required, opened a new log
I20260925 14:53:12.335774 21541 raft_consensus.cc:348] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02029c37bb54988a9aae9a3b83f6520" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39795 } }
I20260925 14:53:12.335914 21541 raft_consensus.cc:374] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:12.335938 21541 raft_consensus.cc:729] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02029c37bb54988a9aae9a3b83f6520, State: Initialized, Role: FOLLOWER
I20260925 14:53:12.336117 21541 consensus_queue.cc:260] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02029c37bb54988a9aae9a3b83f6520" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39795 } }
I20260925 14:53:12.336179 21541 raft_consensus.cc:388] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:12.336200 21541 raft_consensus.cc:482] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:12.336272 21541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:12.336822 21541 raft_consensus.cc:504] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02029c37bb54988a9aae9a3b83f6520" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39795 } }
I20260925 14:53:12.336939 21541 leader_election.cc:302] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b02029c37bb54988a9aae9a3b83f6520; no voters:
I20260925 14:53:12.337129 21541 leader_election.cc:288] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:12.337174 21544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:12.337348 21544 raft_consensus.cc:686] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [term 1 LEADER]: Becoming Leader. State: Replica: b02029c37bb54988a9aae9a3b83f6520, State: Running, Role: LEADER
I20260925 14:53:12.337407 21541 sys_catalog.cc:564] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:12.337553 21544 consensus_queue.cc:237] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02029c37bb54988a9aae9a3b83f6520" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39795 } }
I20260925 14:53:12.338294 21546 sys_catalog.cc:455] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b02029c37bb54988a9aae9a3b83f6520. Latest consensus state: current_term: 1 leader_uuid: "b02029c37bb54988a9aae9a3b83f6520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02029c37bb54988a9aae9a3b83f6520" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39795 } } }
I20260925 14:53:12.338480 21546 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:12.338527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21473
I20260925 14:53:12.338478 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b02029c37bb54988a9aae9a3b83f6520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b02029c37bb54988a9aae9a3b83f6520" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39795 } } }
I20260925 14:53:12.338587 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:12.338654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:12.338766 21553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:12.338845 392 external_mini_cluster.cc:1428] Setting key f29ca1ab475f30692e567f5b0ecfe57a
I20260925 14:53:12.339296 21553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:12.341353 21553 catalog_manager.cc:1347] Generated new cluster ID: da6e200cac2a44ed81f6084307a9422d
I20260925 14:53:12.341408 21553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:12.341892 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.339518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:59173 (local address 127.0.0.1:39795)
0925 14:53:12.340263 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:12.340273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:12.340303 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:53:12.340439 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:12.340444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:12.340497 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:12.340649 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:12.341414 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:12.341418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:12.341419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:12.341424 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:12.341470 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:12.341485 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:12.341526 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:12.341678 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 14:53:12.341723 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":76,"threads_started":1}
I20260925 14:53:12.343088 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:12.345734 21563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:12.344633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:45387 (local address 127.0.0.1:39795)
0925 14:53:12.344673 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:12.344674 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:12.344802 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:53:12.344848 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:12.344849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:12.344854 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:12.344922 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:12.345516 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:12.345517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:12.345518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:12.345522 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:12.345538 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:12.345542 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:12.345604 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:12.345655 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:53:12.345668 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:12.356215 21553 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:12.356424 21553 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:12.371886 21553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b02029c37bb54988a9aae9a3b83f6520: Generated new TSK 0
I20260925 14:53:12.372139 21553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:12.379842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21473
2026-09-25T14:53:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:12 dist-test-slave-z2s5 krb5kdc[21572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:12 dist-test-slave-z2s5 krb5kdc[21572](info): set up 2 sockets
Sep 25 14:53:12 dist-test-slave-z2s5 krb5kdc[21572](info): commencing operation
krb5kdc: starting...
W20260925 14:53:13.426614 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:13 dist-test-slave-z2s5 krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:13.549108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:13.630689 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:13.630977 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:13.631018 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:13.632259 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:13.632349 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:13.632376 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:13.632407 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:13.632439 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:13.634013 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:13.634315 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:13.634599 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:13.639716 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:13.639716 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:13.639716 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:13.640326 21585 server_base.cc:1034] running on GCE node
I20260925 14:53:13.640636 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:13.640995 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:13.642212 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790347993642190 us; error 39 us; skew 500 ppm
Sep 25 14:53:13 dist-test-slave-z2s5 krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347993, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:13.645067 21585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:13.645601 21585 webserver.cc:460] Webserver started at http://127.0.0.1:35847/ using document root <none> and password file <none>
I20260925 14:53:13.645835 21585 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:13.645907 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:13.646070 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:13.647234 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "90763701f81c478e8e71beec334990b9"
format_stamp: "Formatted at 2026-09-25 14:53:13 on dist-test-slave-z2s5"
server_key: "4c922b8c6296807da019d947ce8c2173"
server_key_iv: "4ad2d4f3b880dab0c6c978c1c83c8b3b"
server_key_version: "encryptionkey@0"
I20260925 14:53:13.647614 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "90763701f81c478e8e71beec334990b9"
format_stamp: "Formatted at 2026-09-25 14:53:13 on dist-test-slave-z2s5"
server_key: "4c922b8c6296807da019d947ce8c2173"
server_key_iv: "4ad2d4f3b880dab0c6c978c1c83c8b3b"
server_key_version: "encryptionkey@0"
I20260925 14:53:13.649053 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:13.649914 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:13.650163 21585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:13.650283 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "90763701f81c478e8e71beec334990b9"
format_stamp: "Formatted at 2026-09-25 14:53:13 on dist-test-slave-z2s5"
server_key: "4c922b8c6296807da019d947ce8c2173"
server_key_iv: "4ad2d4f3b880dab0c6c978c1c83c8b3b"
server_key_version: "encryptionkey@0"
I20260925 14:53:13.650365 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:13.661751 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:13.664315 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:13.664489 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:13.668784 21585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42757
I20260925 14:53:13.668821 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42757 every 8 connection(s)
I20260925 14:53:13.669198 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:13.670154 21653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:13.672554 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9: Bootstrap starting.
I20260925 14:53:13.673116 21653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:13.673359 21653 log.cc:824] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:13.674098 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9: No bootstrap required, opened a new log
I20260925 14:53:13.674435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21585
I20260925 14:53:13.674553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:13.674757 392 external_mini_cluster.cc:1428] Setting key 66b801a648bcaa578a33f36de4a60b59
I20260925 14:53:13.676363 21653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90763701f81c478e8e71beec334990b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42757 } }
I20260925 14:53:13.676571 21653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:13.676612 21653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90763701f81c478e8e71beec334990b9, State: Initialized, Role: FOLLOWER
I20260925 14:53:13.676762 21653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90763701f81c478e8e71beec334990b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42757 } }
I20260925 14:53:13.676854 21653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:13.676888 21653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:13.676944 21653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:13.678026 21653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90763701f81c478e8e71beec334990b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42757 } }
I20260925 14:53:13.678124 21653 leader_election.cc:302] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90763701f81c478e8e71beec334990b9; no voters:
I20260925 14:53:13.678326 21653 leader_election.cc:288] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:13.678381 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:13.678577 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [term 1 LEADER]: Becoming Leader. State: Replica: 90763701f81c478e8e71beec334990b9, State: Running, Role: LEADER
I20260925 14:53:13.678643 21653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:13.678758 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90763701f81c478e8e71beec334990b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42757 } }
I20260925 14:53:13.679136 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90763701f81c478e8e71beec334990b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90763701f81c478e8e71beec334990b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42757 } } }
I20260925 14:53:13.679209 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:13.679394 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90763701f81c478e8e71beec334990b9. Latest consensus state: current_term: 1 leader_uuid: "90763701f81c478e8e71beec334990b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90763701f81c478e8e71beec334990b9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42757 } } }
I20260925 14:53:13.679489 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:13.679876 21665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:13.680419 21665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:13.682101 21665 catalog_manager.cc:1347] Generated new cluster ID: 71d27225c26b4b4384590cfa812cd123
I20260925 14:53:13.682155 21665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:13 dist-test-slave-z2s5 krb5kdc[21572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 14:53:13.687283 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.675295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:35387 (local address 127.0.0.1:42757)
0925 14:53:13.675500 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:13.675510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:13.675537 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:53:13.676337 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:13.676345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:13.676399 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:13.676524 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:13.677391 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.678537 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.679671 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.680314 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.680461 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.683452 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:13.683456 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:13.683457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:13.683468 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:13.685416 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:13.685909 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:13.685912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:13.685913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:13.686045 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:13.686343 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:13.686344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:13.686344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:13.686661 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:13.686818 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:13.686970 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 14:53:13.687077 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260925 14:53:13.688333 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:13.697283 21665 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:13.697886 21657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:13.689920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:53087 (local address 127.0.0.1:42757)
0925 14:53:13.689971 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:13.689981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:13.690115 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:53:13.690176 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:13.690177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:13.690188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:13.690239 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:13.690975 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.691717 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.692775 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:13.693090 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:13.693154 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:13.694710 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:13.694712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:13.694712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:13.694720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:13.696400 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:13.696878 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:13.696879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:13.696880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:13.696950 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:13.697197 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:13.697199 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:13.697199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:13.697313 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:13.697448 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:13.697699 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 14:53:13.697769 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:13.698038 21665 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:13.707717 21665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90763701f81c478e8e71beec334990b9: Generated new TSK 0
I20260925 14:53:13.708017 21665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:13.714085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21585
W20260925 14:53:13.718529 21570 connection.cc:535] client connection to 127.0.0.1:42757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42757): Connection reset by peer (error 104)
2026-09-25T14:53:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:13 dist-test-slave-z2s5 krb5kdc[21684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:13 dist-test-slave-z2s5 krb5kdc[21684](info): set up 2 sockets
Sep 25 14:53:13 dist-test-slave-z2s5 krb5kdc[21684](info): commencing operation
krb5kdc: starting...
W20260925 14:53:14.762691 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:14 dist-test-slave-z2s5 krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:14.889133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35207
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:14.971601 21697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:14.971926 21697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:14.971978 21697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:14.973266 21697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:14.973352 21697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:14.973376 21697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:14.973397 21697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:14.973428 21697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:14.975165 21697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35207
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:14.975493 21697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:14.975746 21697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:14.981009 21703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:14.981020 21705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:14.980988 21702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:14.981266 21697 server_base.cc:1034] running on GCE node
I20260925 14:53:14.981585 21697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:14.981943 21697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:14.983156 21697 hybrid_clock.cc:648] HybridClock initialized: now 1790347994983129 us; error 38 us; skew 500 ppm
Sep 25 14:53:14 dist-test-slave-z2s5 krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:14.992969 21697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:14.993535 21697 webserver.cc:460] Webserver started at http://127.0.0.1:34251/ using document root <none> and password file <none>
I20260925 14:53:14.993822 21697 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:14.993904 21697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:14.994065 21697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:14.995287 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "e90f35467ba147d4bd845dcbf2044f8d"
format_stamp: "Formatted at 2026-09-25 14:53:14 on dist-test-slave-z2s5"
server_key: "7dd1f5c1bd7c9a77658272850545ea81"
server_key_iv: "80c02701dc63f86d472f4bc015bc8b64"
server_key_version: "encryptionkey@0"
I20260925 14:53:14.995664 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "e90f35467ba147d4bd845dcbf2044f8d"
format_stamp: "Formatted at 2026-09-25 14:53:14 on dist-test-slave-z2s5"
server_key: "7dd1f5c1bd7c9a77658272850545ea81"
server_key_iv: "80c02701dc63f86d472f4bc015bc8b64"
server_key_version: "encryptionkey@0"
I20260925 14:53:14.997170 21697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:53:14.998121 21712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:14.998447 21697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:14.998570 21697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "e90f35467ba147d4bd845dcbf2044f8d"
format_stamp: "Formatted at 2026-09-25 14:53:14 on dist-test-slave-z2s5"
server_key: "7dd1f5c1bd7c9a77658272850545ea81"
server_key_iv: "80c02701dc63f86d472f4bc015bc8b64"
server_key_version: "encryptionkey@0"
I20260925 14:53:14.998662 21697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:15.011881 21697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:15.014097 21697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:15.014283 21697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:15.018253 21697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35207
I20260925 14:53:15.018306 21764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35207 every 8 connection(s)
I20260925 14:53:15.018719 21697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:15.019752 21765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:15.022096 21765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d: Bootstrap starting.
I20260925 14:53:15.022773 21765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:15.023010 21765 log.cc:824] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:15.023692 21765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d: No bootstrap required, opened a new log
I20260925 14:53:15.024628 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21697
I20260925 14:53:15.024747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:15.024966 392 external_mini_cluster.cc:1428] Setting key 57fbdfeb9756b05d4fa858af2f6fc0ab
I20260925 14:53:15.025911 21765 raft_consensus.cc:348] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f35467ba147d4bd845dcbf2044f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35207 } }
I20260925 14:53:15.026055 21765 raft_consensus.cc:374] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:15.026077 21765 raft_consensus.cc:729] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e90f35467ba147d4bd845dcbf2044f8d, State: Initialized, Role: FOLLOWER
I20260925 14:53:15.026232 21765 consensus_queue.cc:260] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f35467ba147d4bd845dcbf2044f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35207 } }
I20260925 14:53:15.026326 21765 raft_consensus.cc:388] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:15.026364 21765 raft_consensus.cc:482] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:15.026443 21765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:15.027051 21765 raft_consensus.cc:504] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f35467ba147d4bd845dcbf2044f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35207 } }
I20260925 14:53:15.027139 21765 leader_election.cc:302] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e90f35467ba147d4bd845dcbf2044f8d; no voters:
I20260925 14:53:15.027382 21765 leader_election.cc:288] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:15.027405 21770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:15.027516 21770 raft_consensus.cc:686] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [term 1 LEADER]: Becoming Leader. State: Replica: e90f35467ba147d4bd845dcbf2044f8d, State: Running, Role: LEADER
I20260925 14:53:15.027601 21770 consensus_queue.cc:237] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f35467ba147d4bd845dcbf2044f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35207 } }
I20260925 14:53:15.027681 21765 sys_catalog.cc:564] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:15.027938 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e90f35467ba147d4bd845dcbf2044f8d. Latest consensus state: current_term: 1 leader_uuid: "e90f35467ba147d4bd845dcbf2044f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f35467ba147d4bd845dcbf2044f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35207 } } }
I20260925 14:53:15.028016 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:15.027951 21771 sys_catalog.cc:455] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e90f35467ba147d4bd845dcbf2044f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e90f35467ba147d4bd845dcbf2044f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35207 } } }
I20260925 14:53:15.028307 21771 sys_catalog.cc:458] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:15.028427 21775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:15.029006 21775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:15.030805 21775 catalog_manager.cc:1347] Generated new cluster ID: 86c7c510ce5a4212bb7efec44d1250b8
I20260925 14:53:15.030856 21775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:15.031746 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.025559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:51139 (local address 127.0.0.1:35207)
0925 14:53:15.025789 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:15.025798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:15.025825 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:53:15.025928 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:15.025933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:15.025978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:15.026147 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:15.026462 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.028033 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.029204 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.029901 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.030039 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.030866 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:15.030870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:15.030871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:15.031075 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:15.031116 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:15.031131 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:15.031444 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:15.031488 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 14:53:15.031518 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":95,"threads_started":1}
I20260925 14:53:15.035259 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:15.046152 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:15.038779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:33567 (local address 127.0.0.1:35207)
0925 14:53:15.038824 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:15.038826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:15.042610 (+ 3784us) server_negotiation.cc:407] Connection header received
0925 14:53:15.042653 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:15.042654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:15.042668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:15.042741 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:15.042914 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.043674 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.044663 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:15.045014 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:15.045090 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:15.045873 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:15.045874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:15.045875 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:15.045880 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:15.045899 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:15.045904 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:15.045971 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:15.046044 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 14:53:15.046058 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:15.049160 21775 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:15.049930 21775 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:15.063228 21775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e90f35467ba147d4bd845dcbf2044f8d: Generated new TSK 0
I20260925 14:53:15.063469 21775 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:15.067106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21697
W20260925 14:53:15.071918 21682 connection.cc:535] client connection to 127.0.0.1:35207 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35207): Connection reset by peer (error 104)
2026-09-25T14:53:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:15 dist-test-slave-z2s5 krb5kdc[21796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:15 dist-test-slave-z2s5 krb5kdc[21796](info): set up 2 sockets
Sep 25 14:53:15 dist-test-slave-z2s5 krb5kdc[21796](info): commencing operation
krb5kdc: starting...
W20260925 14:53:16.114727 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:16 dist-test-slave-z2s5 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 14:53:16.234104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:16.316552 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:16.316771 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:16.316797 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:16.317960 21809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:16.318015 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:16.318032 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:16.318089 21809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:16.318120 21809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:16.319643 21809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:16.319897 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:16.320137 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:16.325073 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:16.325085 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:16.325182 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:16.325104 21809 server_base.cc:1034] running on GCE node
I20260925 14:53:16.325547 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:16.325820 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:16.326970 21809 hybrid_clock.cc:648] HybridClock initialized: now 1790347996326961 us; error 31 us; skew 500 ppm
Sep 25 14:53:16 dist-test-slave-z2s5 krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:16.329485 21809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 14:53:16.329957 21809 webserver.cc:460] Webserver started at http://127.0.0.1:36103/ using document root <none> and password file <none>
I20260925 14:53:16.330200 21809 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:16.330274 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:16.330473 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:16.331683 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "4435ac50a3044971ae7980c138819d11"
format_stamp: "Formatted at 2026-09-25 14:53:16 on dist-test-slave-z2s5"
server_key: "df77465e2073071c9de6f81589547b96"
server_key_iv: "68c78417a9cd99e5d978bc37d2c8049f"
server_key_version: "encryptionkey@0"
I20260925 14:53:16.332080 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "4435ac50a3044971ae7980c138819d11"
format_stamp: "Formatted at 2026-09-25 14:53:16 on dist-test-slave-z2s5"
server_key: "df77465e2073071c9de6f81589547b96"
server_key_iv: "68c78417a9cd99e5d978bc37d2c8049f"
server_key_version: "encryptionkey@0"
I20260925 14:53:16.333565 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:16.334328 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:16.334558 21809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:16.334666 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "4435ac50a3044971ae7980c138819d11"
format_stamp: "Formatted at 2026-09-25 14:53:16 on dist-test-slave-z2s5"
server_key: "df77465e2073071c9de6f81589547b96"
server_key_iv: "68c78417a9cd99e5d978bc37d2c8049f"
server_key_version: "encryptionkey@0"
I20260925 14:53:16.334743 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:16.338094 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:16.340234 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:16.340399 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:16.344671 21809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41965
I20260925 14:53:16.344704 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41965 every 8 connection(s)
I20260925 14:53:16.345059 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:16.346026 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:16.348351 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11: Bootstrap starting.
I20260925 14:53:16.348869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21809
I20260925 14:53:16.348927 21877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:16.349004 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:16.349193 392 external_mini_cluster.cc:1428] Setting key f55d6c740a592d36b7ccd23fa37e51bc
I20260925 14:53:16.349249 21877 log.cc:824] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:16.350070 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11: No bootstrap required, opened a new log
I20260925 14:53:16.351836 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.349765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:37495 (local address 127.0.0.1:41965)
0925 14:53:16.349941 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:16.349955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:16.349979 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:53:16.350486 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:16.350493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:16.350520 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:16.350673 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:16.351306 (+ 633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:16.351310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:16.351310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:16.351314 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:16.351362 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:16.351372 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:16.351461 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:16.351499 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:53:16.351541 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260925 14:53:16.351912 21877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435ac50a3044971ae7980c138819d11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 14:53:16.352085 21877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:16.352142 21877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4435ac50a3044971ae7980c138819d11, State: Initialized, Role: FOLLOWER
I20260925 14:53:16.352288 21877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435ac50a3044971ae7980c138819d11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 14:53:16.352386 21877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:16.352429 21877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:16.352471 21877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:16.352684 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:16.353081 21877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435ac50a3044971ae7980c138819d11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 14:53:16.353186 21877 leader_election.cc:302] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4435ac50a3044971ae7980c138819d11; no voters:
I20260925 14:53:16.353348 21877 leader_election.cc:288] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:16.353376 21882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:16.353488 21882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [term 1 LEADER]: Becoming Leader. State: Replica: 4435ac50a3044971ae7980c138819d11, State: Running, Role: LEADER
I20260925 14:53:16.353595 21877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:16.353612 21882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435ac50a3044971ae7980c138819d11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 14:53:16.353946 21882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4435ac50a3044971ae7980c138819d11. Latest consensus state: current_term: 1 leader_uuid: "4435ac50a3044971ae7980c138819d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435ac50a3044971ae7980c138819d11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260925 14:53:16.354030 21882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:16.353991 21883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4435ac50a3044971ae7980c138819d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4435ac50a3044971ae7980c138819d11" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260925 14:53:16.354084 21883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:16.354249 21892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:16.354820 21892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:16.359997 21892 catalog_manager.cc:1347] Generated new cluster ID: bb581bc8ad3c4bc8a2fa2c970dec6432
I20260925 14:53:16.360121 21881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:16.358838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.90:48745 (local address 127.0.0.1:41965)
0925 14:53:16.358880 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:16.358883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:16.359013 (+ 130us) server_negotiation.cc:407] Connection header received
0925 14:53:16.359093 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:16.359094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:16.359100 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:16.359150 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:16.359813 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:16.359815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:16.359815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:16.359819 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:16.359837 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:16.359842 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:16.359908 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:16.360002 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:53:16.360017 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:16.360052 21892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:16.381526 21892 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:16.381749 21892 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:16.386274 21892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4435ac50a3044971ae7980c138819d11: Generated new TSK 0
I20260925 14:53:16.386515 21892 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:16.395231 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21809
2026-09-25T14:53:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:16 dist-test-slave-z2s5 krb5kdc[21908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:16 dist-test-slave-z2s5 krb5kdc[21908](info): set up 2 sockets
Sep 25 14:53:16 dist-test-slave-z2s5 krb5kdc[21908](info): commencing operation
krb5kdc: starting...
W20260925 14:53:17.438503 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:17 dist-test-slave-z2s5 krb5kdc[21908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:17.557530 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:17.637957 21921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:17.638177 21921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:17.638200 21921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:17.639510 21921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:17.639556 21921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:17.639570 21921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:17.639583 21921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:17.639595 21921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:17.641093 21921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45659
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:17.641346 21921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:17.641572 21921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:17.646778 21926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:17.646723 21929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:17.646942 21921 server_base.cc:1034] running on GCE node
W20260925 14:53:17.646733 21927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:17.647248 21921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:17.647523 21921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:17.648691 21921 hybrid_clock.cc:648] HybridClock initialized: now 1790347997648669 us; error 35 us; skew 500 ppm
Sep 25 14:53:17 dist-test-slave-z2s5 krb5kdc[21908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347997, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:17.651348 21921 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:17.651834 21921 webserver.cc:460] Webserver started at http://127.0.98.62:37671/ using document root <none> and password file <none>
I20260925 14:53:17.652065 21921 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:17.652135 21921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:17.652285 21921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:17.653369 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "7a53a34328df488bbbb08566a09e882c"
format_stamp: "Formatted at 2026-09-25 14:53:17 on dist-test-slave-z2s5"
server_key: "375edfb919bf6cd98303040fd61bedcf"
server_key_iv: "8231a377e58459443db9fd7b6726ac8d"
server_key_version: "encryptionkey@0"
I20260925 14:53:17.653733 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a53a34328df488bbbb08566a09e882c"
format_stamp: "Formatted at 2026-09-25 14:53:17 on dist-test-slave-z2s5"
server_key: "375edfb919bf6cd98303040fd61bedcf"
server_key_iv: "8231a377e58459443db9fd7b6726ac8d"
server_key_version: "encryptionkey@0"
I20260925 14:53:17.655162 21921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:17.656131 21936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:17.656327 21921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:17.656412 21921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "7a53a34328df488bbbb08566a09e882c"
format_stamp: "Formatted at 2026-09-25 14:53:17 on dist-test-slave-z2s5"
server_key: "375edfb919bf6cd98303040fd61bedcf"
server_key_iv: "8231a377e58459443db9fd7b6726ac8d"
server_key_version: "encryptionkey@0"
I20260925 14:53:17.656482 21921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:17.676167 21921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:17.678258 21921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:17.678457 21921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:17.682547 21921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45659
I20260925 14:53:17.682605 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45659 every 8 connection(s)
I20260925 14:53:17.682976 21921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:17.683828 21989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:17.686121 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c: Bootstrap starting.
I20260925 14:53:17.686771 21989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:17.687007 21989 log.cc:824] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:17.687641 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c: No bootstrap required, opened a new log
I20260925 14:53:17.689186 21989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a53a34328df488bbbb08566a09e882c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:53:17.689319 21989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:17.689345 21989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a53a34328df488bbbb08566a09e882c, State: Initialized, Role: FOLLOWER
I20260925 14:53:17.689482 21989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a53a34328df488bbbb08566a09e882c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:53:17.689581 21989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:17.689605 21989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:17.689639 21989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:17.690104 21989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a53a34328df488bbbb08566a09e882c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:53:17.690182 21989 leader_election.cc:302] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a53a34328df488bbbb08566a09e882c; no voters:
I20260925 14:53:17.690297 21989 leader_election.cc:288] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:17.690351 21992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:17.690539 21992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [term 1 LEADER]: Becoming Leader. State: Replica: 7a53a34328df488bbbb08566a09e882c, State: Running, Role: LEADER
I20260925 14:53:17.690589 21989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:17.690623 21992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a53a34328df488bbbb08566a09e882c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } }
I20260925 14:53:17.691039 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a53a34328df488bbbb08566a09e882c. Latest consensus state: current_term: 1 leader_uuid: "7a53a34328df488bbbb08566a09e882c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a53a34328df488bbbb08566a09e882c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } }
I20260925 14:53:17.691120 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:17.691274 21993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a53a34328df488bbbb08566a09e882c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a53a34328df488bbbb08566a09e882c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45659 } } }
I20260925 14:53:17.691376 21993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:17.691728 22000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:17.692200 22000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:17.692605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 21921
I20260925 14:53:17.692715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:17.692902 392 external_mini_cluster.cc:1428] Setting key 1d74f593339546f3a9292e25fc31c7e5
I20260925 14:53:17.693876 22000 catalog_manager.cc:1347] Generated new cluster ID: 065f9044956b4773bf189311eda53435
I20260925 14:53:17.693928 22000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:17 dist-test-slave-z2s5 krb5kdc[21908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:17.704984 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.693361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36333 (local address 127.0.98.62:45659)
0925 14:53:17.693594 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:17.693604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:17.693638 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:53:17.693756 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:17.693819 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:17.693877 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:17.694023 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:17.694969 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.696391 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.697499 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.698115 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.698209 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.701162 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:17.701167 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:17.701168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:17.701181 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:17.703250 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:17.703693 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:17.703696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:17.703697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:17.703808 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:17.704047 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:17.704048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:17.704048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:17.704368 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:17.704550 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:17.704642 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:53:17.704743 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":84,"threads_started":1}
I20260925 14:53:17.706017 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:17.714761 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:17.707617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45005 (local address 127.0.98.62:45659)
0925 14:53:17.707652 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:17.707654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:17.707663 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:17.707720 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:17.707721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:17.707730 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:17.707779 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:17.708371 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.709127 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.710088 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:17.710502 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:17.710572 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:17.711870 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:17.711871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:17.711871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:17.711877 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:17.713433 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:17.713805 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:17.713805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:17.713806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:17.713863 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:17.714079 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:17.714080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:17.714080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:17.714245 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:17.714375 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:17.714505 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:53:17.714568 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:17.715898 22000 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:17.716372 22000 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:17.725446 22000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a53a34328df488bbbb08566a09e882c: Generated new TSK 0
I20260925 14:53:17.725661 22000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:17.728513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 21921
W20260925 14:53:17.732988 21906 connection.cc:535] client connection to 127.0.98.62:45659 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45659): Connection reset by peer (error 104)
2026-09-25T14:53:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:17 dist-test-slave-z2s5 krb5kdc[22020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:17 dist-test-slave-z2s5 krb5kdc[22020](info): set up 2 sockets
Sep 25 14:53:17 dist-test-slave-z2s5 krb5kdc[22020](info): commencing operation
krb5kdc: starting...
W20260925 14:53:18.786782 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:18 dist-test-slave-z2s5 krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:18.905129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:18.984256 22033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:18.984508 22033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:18.984548 22033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:18.985678 22033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:18.985751 22033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:18.985775 22033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:18.985813 22033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:18.985837 22033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:18.987336 22033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:18.987605 22033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:18.987828 22033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:18.992472 22039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:18.992468 22041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:18.992683 22033 server_base.cc:1034] running on GCE node
W20260925 14:53:18.992532 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:18.992988 22033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:18.993218 22033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:18.994359 22033 hybrid_clock.cc:648] HybridClock initialized: now 1790347998994344 us; error 27 us; skew 500 ppm
Sep 25 14:53:18 dist-test-slave-z2s5 krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790347998, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:18.996793 22033 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:18.997231 22033 webserver.cc:460] Webserver started at http://127.0.98.62:40495/ using document root <none> and password file <none>
I20260925 14:53:18.997454 22033 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:18.997517 22033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:18.997655 22033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:18.998804 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "635d6632e35f4c859cf9e2d7b70ca210"
format_stamp: "Formatted at 2026-09-25 14:53:18 on dist-test-slave-z2s5"
server_key: "d5dd0d142669eb2f7e871b77d7f896f1"
server_key_iv: "7cd58afcc1fab034838e1d51bec3001d"
server_key_version: "encryptionkey@0"
I20260925 14:53:18.999147 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "635d6632e35f4c859cf9e2d7b70ca210"
format_stamp: "Formatted at 2026-09-25 14:53:18 on dist-test-slave-z2s5"
server_key: "d5dd0d142669eb2f7e871b77d7f896f1"
server_key_iv: "7cd58afcc1fab034838e1d51bec3001d"
server_key_version: "encryptionkey@0"
I20260925 14:53:19.000438 22033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:19.001140 22048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:19.001317 22033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:19.001415 22033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "635d6632e35f4c859cf9e2d7b70ca210"
format_stamp: "Formatted at 2026-09-25 14:53:18 on dist-test-slave-z2s5"
server_key: "d5dd0d142669eb2f7e871b77d7f896f1"
server_key_iv: "7cd58afcc1fab034838e1d51bec3001d"
server_key_version: "encryptionkey@0"
I20260925 14:53:19.001484 22033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:19.016127 22033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:19.018529 22033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:19.018695 22033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:19.022825 22033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38737
I20260925 14:53:19.022858 22100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38737 every 8 connection(s)
I20260925 14:53:19.023212 22033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:19.024057 22101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:19.026368 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210: Bootstrap starting.
I20260925 14:53:19.026970 22101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:19.027254 22101 log.cc:824] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:19.028028 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210: No bootstrap required, opened a new log
I20260925 14:53:19.029757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22033
I20260925 14:53:19.029676 22101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635d6632e35f4c859cf9e2d7b70ca210" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38737 } }
I20260925 14:53:19.029871 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:19.029857 22101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:19.029901 22101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635d6632e35f4c859cf9e2d7b70ca210, State: Initialized, Role: FOLLOWER
I20260925 14:53:19.030069 392 external_mini_cluster.cc:1428] Setting key fff7273e0c43c10554ad315dfdd2bcdb
I20260925 14:53:19.030064 22101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635d6632e35f4c859cf9e2d7b70ca210" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38737 } }
I20260925 14:53:19.030143 22101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:19.030185 22101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:19.030249 22101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:19.030989 22101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635d6632e35f4c859cf9e2d7b70ca210" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38737 } }
I20260925 14:53:19.031117 22101 leader_election.cc:302] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 635d6632e35f4c859cf9e2d7b70ca210; no voters:
I20260925 14:53:19.031279 22101 leader_election.cc:288] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:19.031322 22106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:19.031462 22106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [term 1 LEADER]: Becoming Leader. State: Replica: 635d6632e35f4c859cf9e2d7b70ca210, State: Running, Role: LEADER
I20260925 14:53:19.031549 22101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:19.031550 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635d6632e35f4c859cf9e2d7b70ca210" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38737 } }
I20260925 14:53:19.032016 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 635d6632e35f4c859cf9e2d7b70ca210. Latest consensus state: current_term: 1 leader_uuid: "635d6632e35f4c859cf9e2d7b70ca210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635d6632e35f4c859cf9e2d7b70ca210" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38737 } } }
I20260925 14:53:19.032109 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:19.032027 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "635d6632e35f4c859cf9e2d7b70ca210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "635d6632e35f4c859cf9e2d7b70ca210" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38737 } } }
I20260925 14:53:19.032292 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:19.032316 22114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:19.032922 22114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:19.034871 22114 catalog_manager.cc:1347] Generated new cluster ID: 8afc503cc18e44429e87d75a3ed6a83b
I20260925 14:53:19.034924 22114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:19.036692 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.030613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44929 (local address 127.0.98.62:38737)
0925 14:53:19.031221 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:19.031234 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:19.031281 (+ 47us) server_negotiation.cc:407] Connection header received
0925 14:53:19.031403 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:19.031412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:19.031477 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:19.031878 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:19.031978 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.033512 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.034598 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.035188 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.035289 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.036034 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:19.036038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:19.036038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:19.036232 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:19.036270 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:19.036283 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:19.036361 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:19.036476 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:53:19.036501 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":515,"threads_started":1}
I20260925 14:53:19.037635 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:19.042675 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:19.039200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50473 (local address 127.0.98.62:38737)
0925 14:53:19.039258 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:19.039260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:19.039316 (+ 56us) server_negotiation.cc:407] Connection header received
0925 14:53:19.039394 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:19.039395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:19.039404 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:19.039455 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:19.039594 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.040301 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.041329 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:19.041635 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:19.041696 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:19.042425 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:19.042426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:19.042426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:19.042430 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:19.042445 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:19.042449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:19.042500 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:19.042592 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:53:19.042603 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:19.057902 22114 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:19.058494 22114 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:19.061260 22114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 635d6632e35f4c859cf9e2d7b70ca210: Generated new TSK 0
I20260925 14:53:19.061448 22114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:19.068789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22033
W20260925 14:53:19.074510 22018 connection.cc:535] client connection to 127.0.98.62:38737 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38737): Connection reset by peer (error 104)
2026-09-25T14:53:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:19 dist-test-slave-z2s5 krb5kdc[22132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:19 dist-test-slave-z2s5 krb5kdc[22132](info): set up 2 sockets
Sep 25 14:53:19 dist-test-slave-z2s5 krb5kdc[22132](info): commencing operation
krb5kdc: starting...
W20260925 14:53:20.222710 392 mini_kdc.cc:121] Time spent starting KDC: real 1.134s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:20 dist-test-slave-z2s5 krb5kdc[22132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:20.341741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42553
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:20.428495 22145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:20.428763 22145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:20.428804 22145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:20.429986 22145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:20.430054 22145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:20.430081 22145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:20.430116 22145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:20.430140 22145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:20.431679 22145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:20.431955 22145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:20.432176 22145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:20.437027 22153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:20.437093 22145 server_base.cc:1034] running on GCE node
W20260925 14:53:20.437018 22150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:20.437018 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:20.437549 22145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:20.437876 22145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:20.439055 22145 hybrid_clock.cc:648] HybridClock initialized: now 1790348000439046 us; error 37 us; skew 500 ppm
Sep 25 14:53:20 dist-test-slave-z2s5 krb5kdc[22132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348000, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:20.448624 22145 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:20.449108 22145 webserver.cc:460] Webserver started at http://127.0.98.62:39599/ using document root <none> and password file <none>
I20260925 14:53:20.449318 22145 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:20.449368 22145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:20.449478 22145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:20.450692 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "5b94f98522d146c1ae57e238eaed4067"
format_stamp: "Formatted at 2026-09-25 14:53:20 on dist-test-slave-z2s5"
server_key: "debf850e0031f0607ab1e45620b198cd"
server_key_iv: "5acae08292d1d4b302311145b71c83c8"
server_key_version: "encryptionkey@0"
I20260925 14:53:20.451033 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "5b94f98522d146c1ae57e238eaed4067"
format_stamp: "Formatted at 2026-09-25 14:53:20 on dist-test-slave-z2s5"
server_key: "debf850e0031f0607ab1e45620b198cd"
server_key_iv: "5acae08292d1d4b302311145b71c83c8"
server_key_version: "encryptionkey@0"
I20260925 14:53:20.452430 22145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:20.453207 22160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:20.453450 22145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:20.453544 22145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "5b94f98522d146c1ae57e238eaed4067"
format_stamp: "Formatted at 2026-09-25 14:53:20 on dist-test-slave-z2s5"
server_key: "debf850e0031f0607ab1e45620b198cd"
server_key_iv: "5acae08292d1d4b302311145b71c83c8"
server_key_version: "encryptionkey@0"
I20260925 14:53:20.453629 22145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:20.457001 22145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:20.459105 22145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:20.459242 22145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:20.463277 22145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42553
I20260925 14:53:20.463333 22212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42553 every 8 connection(s)
I20260925 14:53:20.463680 22145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:20.464684 22213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:20.467011 22213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067: Bootstrap starting.
I20260925 14:53:20.467590 22213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:20.467887 22213 log.cc:824] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:20.468580 22213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067: No bootstrap required, opened a new log
I20260925 14:53:20.470208 22213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b94f98522d146c1ae57e238eaed4067" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260925 14:53:20.470368 22213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:20.470439 22213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b94f98522d146c1ae57e238eaed4067, State: Initialized, Role: FOLLOWER
I20260925 14:53:20.470599 22213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b94f98522d146c1ae57e238eaed4067" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260925 14:53:20.470681 22213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:20.470726 22213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:20.470777 22213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:20.471004 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22145
I20260925 14:53:20.471119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:20.471308 392 external_mini_cluster.cc:1428] Setting key f495af242a1bda4a509bce7c0a9bb2e7
I20260925 14:53:20.471451 22213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b94f98522d146c1ae57e238eaed4067" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260925 14:53:20.471560 22213 leader_election.cc:302] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b94f98522d146c1ae57e238eaed4067; no voters:
I20260925 14:53:20.471717 22213 leader_election.cc:288] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:20.471764 22216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:20.471911 22216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [term 1 LEADER]: Becoming Leader. State: Replica: 5b94f98522d146c1ae57e238eaed4067, State: Running, Role: LEADER
I20260925 14:53:20.472064 22216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b94f98522d146c1ae57e238eaed4067" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260925 14:53:20.472450 22216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b94f98522d146c1ae57e238eaed4067. Latest consensus state: current_term: 1 leader_uuid: "5b94f98522d146c1ae57e238eaed4067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b94f98522d146c1ae57e238eaed4067" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } } }
I20260925 14:53:20.472565 22216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:20.472630 22218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b94f98522d146c1ae57e238eaed4067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b94f98522d146c1ae57e238eaed4067" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } } }
I20260925 14:53:20.473022 22218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:20.472055 22213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:20.473935 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.471813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34565 (local address 127.0.98.62:42553)
0925 14:53:20.472136 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:20.472146 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:20.472182 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:53:20.472295 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:20.472302 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:20.472333 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:20.472492 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:20.473432 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:20.473436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:20.473436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:20.473441 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:20.473493 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:20.473509 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:20.473595 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:20.473647 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:53:20.473689 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":93,"threads_started":1}
I20260925 14:53:20.472867 22222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:20.474887 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:20.474967 22222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:20.476745 22222 catalog_manager.cc:1347] Generated new cluster ID: 6178a67063fe4a5780dda6c48135a3d7
I20260925 14:53:20.476786 22222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:20.483966 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:20.482747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34637 (local address 127.0.98.62:42553)
0925 14:53:20.482786 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:20.482788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:20.482906 (+ 118us) server_negotiation.cc:407] Connection header received
0925 14:53:20.482977 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:20.482978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:20.482983 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:20.483055 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:20.483728 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:20.483729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:20.483729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:20.483733 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:20.483747 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:20.483751 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:20.483808 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:20.483874 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:53:20.483887 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:20.489795 22222 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:20.489961 22222 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:20.492447 22222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b94f98522d146c1ae57e238eaed4067: Generated new TSK 0
I20260925 14:53:20.492646 22222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:20.494639 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22145
2026-09-25T14:53:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:20 dist-test-slave-z2s5 krb5kdc[22244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:20 dist-test-slave-z2s5 krb5kdc[22244](info): set up 2 sockets
Sep 25 14:53:20 dist-test-slave-z2s5 krb5kdc[22244](info): commencing operation
krb5kdc: starting...
W20260925 14:53:21.531126 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:21 dist-test-slave-z2s5 krb5kdc[22244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:21.649407 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:21.731094 22257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:21.731299 22257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:21.731323 22257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:21.732393 22257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:21.732429 22257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:21.732441 22257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:21.732455 22257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:21.732466 22257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:21.733934 22257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:21.734129 22257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:21.734311 22257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:21.739149 22265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:21.739154 22263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:21.739162 22262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:21.739223 22257 server_base.cc:1034] running on GCE node
I20260925 14:53:21.739606 22257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:21.739874 22257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:21.741025 22257 hybrid_clock.cc:648] HybridClock initialized: now 1790348001741004 us; error 46 us; skew 500 ppm
Sep 25 14:53:21 dist-test-slave-z2s5 krb5kdc[22244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:21.743618 22257 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:21.744148 22257 webserver.cc:460] Webserver started at http://127.0.98.62:34951/ using document root <none> and password file <none>
I20260925 14:53:21.744387 22257 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:21.744454 22257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:21.744599 22257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:21.745656 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "a3d2271de38a4ae59d842b8f326268e1"
format_stamp: "Formatted at 2026-09-25 14:53:21 on dist-test-slave-z2s5"
server_key: "4df8dad0993e59d4a38430c47bdc8223"
server_key_iv: "64f9b1c0876b76c9885bc847de0b6710"
server_key_version: "encryptionkey@0"
I20260925 14:53:21.746016 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "a3d2271de38a4ae59d842b8f326268e1"
format_stamp: "Formatted at 2026-09-25 14:53:21 on dist-test-slave-z2s5"
server_key: "4df8dad0993e59d4a38430c47bdc8223"
server_key_iv: "64f9b1c0876b76c9885bc847de0b6710"
server_key_version: "encryptionkey@0"
I20260925 14:53:21.747442 22257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:21.748342 22272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:21.748580 22257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:21.748675 22257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "a3d2271de38a4ae59d842b8f326268e1"
format_stamp: "Formatted at 2026-09-25 14:53:21 on dist-test-slave-z2s5"
server_key: "4df8dad0993e59d4a38430c47bdc8223"
server_key_iv: "64f9b1c0876b76c9885bc847de0b6710"
server_key_version: "encryptionkey@0"
I20260925 14:53:21.748750 22257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:21.758544 22257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:21.760891 22257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:21.761059 22257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:21.765090 22257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32883
I20260925 14:53:21.765125 22324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32883 every 8 connection(s)
I20260925 14:53:21.765475 22257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:21.766330 22325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:21.768683 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1: Bootstrap starting.
I20260925 14:53:21.769279 22325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:21.769557 22325 log.cc:824] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:21.770262 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1: No bootstrap required, opened a new log
I20260925 14:53:21.771968 22325 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d2271de38a4ae59d842b8f326268e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260925 14:53:21.772104 22325 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:21.772126 22325 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3d2271de38a4ae59d842b8f326268e1, State: Initialized, Role: FOLLOWER
I20260925 14:53:21.772276 22325 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d2271de38a4ae59d842b8f326268e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260925 14:53:21.772339 22325 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:21.772377 22325 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:21.772426 22325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:21.772933 22325 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d2271de38a4ae59d842b8f326268e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260925 14:53:21.773041 22325 leader_election.cc:302] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3d2271de38a4ae59d842b8f326268e1; no voters:
I20260925 14:53:21.773205 22325 leader_election.cc:288] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:21.773247 22328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:21.773427 22328 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [term 1 LEADER]: Becoming Leader. State: Replica: a3d2271de38a4ae59d842b8f326268e1, State: Running, Role: LEADER
I20260925 14:53:21.773458 22325 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:21.773571 22328 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d2271de38a4ae59d842b8f326268e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } }
I20260925 14:53:21.773945 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3d2271de38a4ae59d842b8f326268e1. Latest consensus state: current_term: 1 leader_uuid: "a3d2271de38a4ae59d842b8f326268e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d2271de38a4ae59d842b8f326268e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } } }
I20260925 14:53:21.773943 22329 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3d2271de38a4ae59d842b8f326268e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3d2271de38a4ae59d842b8f326268e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32883 } } }
I20260925 14:53:21.774070 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:21.774134 22329 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:21.774318 22332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:21.774456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22257
I20260925 14:53:21.774585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:21.774789 392 external_mini_cluster.cc:1428] Setting key 67d2f0fab31473fe89ae1aee51f6a809
I20260925 14:53:21.774902 22332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:21.776773 22332 catalog_manager.cc:1347] Generated new cluster ID: a58e61b68e924c558b6912710d878048
I20260925 14:53:21.776827 22332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:21 dist-test-slave-z2s5 krb5kdc[22244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:21.782795 22332 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:21.783269 22332 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:21.785306 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:21.775444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49753 (local address 127.0.98.62:32883)
0925 14:53:21.775984 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:21.775992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:21.776025 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:53:21.776134 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:21.776141 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:21.776195 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:21.776352 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:21.777033 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.777937 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.778783 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.779259 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.779393 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.781638 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:21.781641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:21.781642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:21.781654 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:21.783732 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:21.784109 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:21.784112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:21.784112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:21.784225 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:21.784426 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:21.784427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:21.784427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:21.784711 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:21.784886 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:21.784956 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:53:21.785108 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":490,"thread_start_us":78,"threads_started":1}
I20260925 14:53:21.786150 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:21.789969 22332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3d2271de38a4ae59d842b8f326268e1: Generated new TSK 0
I20260925 14:53:21.790131 22332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:21.794220 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:21.787686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44429 (local address 127.0.98.62:32883)
0925 14:53:21.787720 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:21.787722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:21.787745 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:53:21.787790 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:21.787790 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:21.787799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:21.787844 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:21.788436 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.789206 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.790185 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:21.790566 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:21.790633 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:21.791733 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:21.791735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:21.791735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:21.791742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:21.793222 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:21.793539 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:21.793540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:21.793540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:21.793596 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:21.793763 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:21.793764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:21.793764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:21.793884 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:21.793978 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:21.794057 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:53:21.794132 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:21.796348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22257
2026-09-25T14:53:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:21 dist-test-slave-z2s5 krb5kdc[22356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:21 dist-test-slave-z2s5 krb5kdc[22356](info): set up 2 sockets
Sep 25 14:53:21 dist-test-slave-z2s5 krb5kdc[22356](info): commencing operation
krb5kdc: starting...
W20260925 14:53:22.834596 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:22 dist-test-slave-z2s5 krb5kdc[22356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:22.954212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43887
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:23.033675 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:23.033931 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:23.033970 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:23.035151 22369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:23.035224 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:23.035248 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:23.035269 22369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:23.035297 22369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:23.036773 22369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:23.037047 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:23.037276 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:23.042125 22375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:23.042217 22374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:23.042125 22377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:23.042198 22369 server_base.cc:1034] running on GCE node
I20260925 14:53:23.042636 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:23.042887 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:23.044050 22369 hybrid_clock.cc:648] HybridClock initialized: now 1790348003044028 us; error 36 us; skew 500 ppm
Sep 25 14:53:23 dist-test-slave-z2s5 krb5kdc[22356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348003, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:23.046819 22369 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:23.047246 22369 webserver.cc:460] Webserver started at http://127.0.98.62:35567/ using document root <none> and password file <none>
I20260925 14:53:23.047444 22369 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:23.047490 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:23.047592 22369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:23.048717 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "2b88257478be4624878bd0d5c416ca40"
format_stamp: "Formatted at 2026-09-25 14:53:23 on dist-test-slave-z2s5"
server_key: "0cb50ebc09b96bedba7750ade57c47f2"
server_key_iv: "7118cc1cb7ccc546b2a964cb5f0eadd2"
server_key_version: "encryptionkey@0"
I20260925 14:53:23.049041 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b88257478be4624878bd0d5c416ca40"
format_stamp: "Formatted at 2026-09-25 14:53:23 on dist-test-slave-z2s5"
server_key: "0cb50ebc09b96bedba7750ade57c47f2"
server_key_iv: "7118cc1cb7ccc546b2a964cb5f0eadd2"
server_key_version: "encryptionkey@0"
I20260925 14:53:23.050380 22369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:53:23.051254 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:23.051402 22369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:23.051489 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "2b88257478be4624878bd0d5c416ca40"
format_stamp: "Formatted at 2026-09-25 14:53:23 on dist-test-slave-z2s5"
server_key: "0cb50ebc09b96bedba7750ade57c47f2"
server_key_iv: "7118cc1cb7ccc546b2a964cb5f0eadd2"
server_key_version: "encryptionkey@0"
I20260925 14:53:23.051545 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:23.070883 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:23.072897 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:23.073009 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:23.076776 22369 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43887
I20260925 14:53:23.076858 22436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43887 every 8 connection(s)
I20260925 14:53:23.077200 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:23.078073 22437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:23.079077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22369
I20260925 14:53:23.079166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:23.079368 392 external_mini_cluster.cc:1428] Setting key 269f2496239341c7905d7a87cf566dd8
I20260925 14:53:23.081099 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40: Bootstrap starting.
I20260925 14:53:23.081691 22437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:23.081986 22437 log.cc:824] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:23.082670 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40: No bootstrap required, opened a new log
I20260925 14:53:23.084384 22437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88257478be4624878bd0d5c416ca40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260925 14:53:23.084533 22437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:23.084611 22437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b88257478be4624878bd0d5c416ca40, State: Initialized, Role: FOLLOWER
I20260925 14:53:23.084775 22437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88257478be4624878bd0d5c416ca40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260925 14:53:23.084875 22437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:23.084915 22437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:23.084935 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:23.079897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60011 (local address 127.0.98.62:43887)
0925 14:53:23.080086 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:23.080096 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:23.080128 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:53:23.080817 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:23.080826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:23.080898 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:23.081049 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:23.081238 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.082228 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.083105 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.083530 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.083656 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.084296 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:23.084300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:23.084300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:23.084469 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:23.084499 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:23.084510 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:23.084583 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:23.084658 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 14:53:23.084720 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
I20260925 14:53:23.084964 22437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:23.085646 22437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88257478be4624878bd0d5c416ca40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260925 14:53:23.085750 22437 leader_election.cc:302] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b88257478be4624878bd0d5c416ca40; no voters:
I20260925 14:53:23.085923 22437 leader_election.cc:288] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:23.085922 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:23.085960 22442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:23.086102 22442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [term 1 LEADER]: Becoming Leader. State: Replica: 2b88257478be4624878bd0d5c416ca40, State: Running, Role: LEADER
I20260925 14:53:23.086192 22437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:23.086226 22442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88257478be4624878bd0d5c416ca40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } }
I20260925 14:53:23.086644 22442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b88257478be4624878bd0d5c416ca40. Latest consensus state: current_term: 1 leader_uuid: "2b88257478be4624878bd0d5c416ca40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88257478be4624878bd0d5c416ca40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } } }
I20260925 14:53:23.086658 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b88257478be4624878bd0d5c416ca40" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b88257478be4624878bd0d5c416ca40" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43887 } } }
I20260925 14:53:23.086807 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:23.087000 22442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:23.087399 22451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:23.087901 22451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:23.089304 22451 catalog_manager.cc:1347] Generated new cluster ID: c0b686d63b194d87a96ad6ab83a7ce2a
I20260925 14:53:23.089345 22451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:23.093938 22440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:23.087871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39119 (local address 127.0.98.62:43887)
0925 14:53:23.087924 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:23.087926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:23.090550 (+ 2624us) server_negotiation.cc:407] Connection header received
0925 14:53:23.090649 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:23.090651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:23.090663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:23.090731 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:23.090870 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.091621 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.092593 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:23.092909 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:23.092988 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:23.093658 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:23.093659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:23.093659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:23.093664 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:23.093680 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:23.093685 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:23.093753 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:23.093812 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:53:23.093843 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:53:23.126780 22451 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:23.127352 22451 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:23.130720 22451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b88257478be4624878bd0d5c416ca40: Generated new TSK 0
I20260925 14:53:23.130915 22451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:23.136116 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22369
2026-09-25T14:53:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:23 dist-test-slave-z2s5 krb5kdc[22468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:23 dist-test-slave-z2s5 krb5kdc[22468](info): set up 2 sockets
Sep 25 14:53:23 dist-test-slave-z2s5 krb5kdc[22468](info): commencing operation
krb5kdc: starting...
W20260925 14:53:24.158715 392 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:24 dist-test-slave-z2s5 krb5kdc[22468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:24.277832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41713
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:24.357800 22481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:24.358039 22481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:24.358078 22481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:24.359231 22481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:24.359297 22481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:24.359321 22481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:24.359344 22481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:24.359380 22481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:24.360814 22481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:24.361083 22481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:24.361290 22481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:24.365938 22489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:24.365912 22486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:24.365942 22487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:24.366389 22481 server_base.cc:1034] running on GCE node
I20260925 14:53:24.366644 22481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:24.366914 22481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:24.368072 22481 hybrid_clock.cc:648] HybridClock initialized: now 1790348004368065 us; error 36 us; skew 500 ppm
Sep 25 14:53:24 dist-test-slave-z2s5 krb5kdc[22468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348004, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:24.370453 22481 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:24.370846 22481 webserver.cc:460] Webserver started at http://127.0.98.62:40403/ using document root <none> and password file <none>
I20260925 14:53:24.371084 22481 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:24.371151 22481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:24.371280 22481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:24.372332 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b"
format_stamp: "Formatted at 2026-09-25 14:53:24 on dist-test-slave-z2s5"
server_key: "d8f879d1f267132edd92e4430b987e48"
server_key_iv: "43c1b1282805b5d766e0abf06f648361"
server_key_version: "encryptionkey@0"
I20260925 14:53:24.372676 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b"
format_stamp: "Formatted at 2026-09-25 14:53:24 on dist-test-slave-z2s5"
server_key: "d8f879d1f267132edd92e4430b987e48"
server_key_iv: "43c1b1282805b5d766e0abf06f648361"
server_key_version: "encryptionkey@0"
I20260925 14:53:24.373977 22481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:24.374738 22496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:24.375011 22481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:53:24.375115 22481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b"
format_stamp: "Formatted at 2026-09-25 14:53:24 on dist-test-slave-z2s5"
server_key: "d8f879d1f267132edd92e4430b987e48"
server_key_iv: "43c1b1282805b5d766e0abf06f648361"
server_key_version: "encryptionkey@0"
I20260925 14:53:24.375183 22481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:24.378259 22481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:24.380084 22481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:24.380234 22481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:24.384064 22481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41713
I20260925 14:53:24.384110 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41713 every 8 connection(s)
I20260925 14:53:24.384433 22481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:24.385426 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:24.387785 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b: Bootstrap starting.
I20260925 14:53:24.388337 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:24.388665 22549 log.cc:824] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:24.389519 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b: No bootstrap required, opened a new log
I20260925 14:53:24.392002 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260925 14:53:24.392189 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:53:24.392227 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c617c4c24e7f4f7a8bd43469d4d52c1b, State: Initialized, Role: FOLLOWER
I20260925 14:53:24.392323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22481
I20260925 14:53:24.392366 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260925 14:53:24.392436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:24.392444 22549 raft_consensus.cc:388] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:53:24.392472 22549 raft_consensus.cc:482] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:53:24.392509 22549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:24.392642 392 external_mini_cluster.cc:1428] Setting key f2d253fbd84d3904f7b8ce6921b25462
I20260925 14:53:24.393181 22549 raft_consensus.cc:504] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260925 14:53:24.393283 22549 leader_election.cc:302] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c617c4c24e7f4f7a8bd43469d4d52c1b; no voters:
I20260925 14:53:24.393450 22549 leader_election.cc:288] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:53:24.393541 22554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:24.393659 22554 raft_consensus.cc:686] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [term 1 LEADER]: Becoming Leader. State: Replica: c617c4c24e7f4f7a8bd43469d4d52c1b, State: Running, Role: LEADER
I20260925 14:53:24.393687 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:24.393733 22554 consensus_queue.cc:237] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } }
I20260925 14:53:24.394052 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } } }
I20260925 14:53:24.394090 22555 sys_catalog.cc:455] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c617c4c24e7f4f7a8bd43469d4d52c1b. Latest consensus state: current_term: 1 leader_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c617c4c24e7f4f7a8bd43469d4d52c1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41713 } } }
I20260925 14:53:24.394140 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:24.394164 22555 sys_catalog.cc:458] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:24.394901 22558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:24.395404 22558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:24.396910 22558 catalog_manager.cc:1347] Generated new cluster ID: dab695ff23a54b30950e42542f236232
I20260925 14:53:24.396970 22558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:24.397066 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:24.393084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49811 (local address 127.0.98.62:41713)
0925 14:53:24.393259 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:24.393268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:24.393460 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:53:24.393614 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:24.393623 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:24.393666 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:24.393812 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:24.396553 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:24.396555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:24.396556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:24.396560 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:24.396633 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:24.396645 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:24.396721 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:24.396795 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 14:53:24.396839 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260925 14:53:24.397953 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:24.400483 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:24.399352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33343 (local address 127.0.98.62:41713)
0925 14:53:24.399489 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:24.399491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:24.399501 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:24.399527 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:24.399528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:24.399533 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:24.399604 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:24.400228 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:24.400229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:24.400229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:53:24.400234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:24.400250 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:53:24.400255 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:24.400285 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:24.400386 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 14:53:24.400401 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 14:53:24.411576 22558 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:24.411769 22558 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:24.420936 22558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c617c4c24e7f4f7a8bd43469d4d52c1b: Generated new TSK 0
I20260925 14:53:24.421145 22558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:24.429332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22481
2026-09-25T14:53:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1293 ms)
[----------] 18 tests from AuthTokenIssuingTest (24056 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:24 dist-test-slave-z2s5 krb5kdc[22580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:24 dist-test-slave-z2s5 krb5kdc[22580](info): set up 2 sockets
Sep 25 14:53:24 dist-test-slave-z2s5 krb5kdc[22580](info): commencing operation
krb5kdc: starting...
W20260925 14:53:25.462649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:25.581130 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46647,127.0.98.61:39757,127.0.98.60:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:25.660353 22593 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:53:25.660602 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:25.660631 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:25.660672 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:25.661799 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:25.661868 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:25.661895 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:25.661926 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:25.661947 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:25.663417 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46647,127.0.98.61:39757,127.0.98.60:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:25.663691 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:25.663913 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:25.668709 22599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:25.668715 22598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:25.668730 22601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:25.668970 22593 server_base.cc:1034] running on GCE node
I20260925 14:53:25.669183 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:25.669431 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:25.670569 22593 hybrid_clock.cc:648] HybridClock initialized: now 1790348005670561 us; error 35 us; skew 500 ppm
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:25.676234 22593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:25.676692 22593 webserver.cc:460] Webserver started at http://127.0.98.62:34505/ using document root <none> and password file <none>
I20260925 14:53:25.676882 22593 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:25.676923 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:25.677009 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:25.678025 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "e61c305f741c43c780ebc9a23df79d7f"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "aad45af5ff3bf1e3dcd010b6728a0d26"
server_key_iv: "fedfee4c6d755dd931343f6d347d014c"
server_key_version: "encryptionkey@0"
I20260925 14:53:25.678323 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "e61c305f741c43c780ebc9a23df79d7f"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "aad45af5ff3bf1e3dcd010b6728a0d26"
server_key_iv: "fedfee4c6d755dd931343f6d347d014c"
server_key_version: "encryptionkey@0"
I20260925 14:53:25.679651 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:53:25.680343 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:25.680532 22593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:25.680631 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "e61c305f741c43c780ebc9a23df79d7f"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "aad45af5ff3bf1e3dcd010b6728a0d26"
server_key_iv: "fedfee4c6d755dd931343f6d347d014c"
server_key_version: "encryptionkey@0"
I20260925 14:53:25.680702 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:25.699930 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:25.701915 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:25.702075 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:25.706169 22593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46647
I20260925 14:53:25.706205 22660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46647 every 8 connection(s)
I20260925 14:53:25.706622 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:25.707597 22661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:25.709062 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:25.715999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22593
I20260925 14:53:25.716142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:25.716416 392 external_mini_cluster.cc:1428] Setting key 80fe70dfd511dbc9f6fa3a9c58a0270c
I20260925 14:53:25.718983 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.709678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46647 (local address 127.0.0.1:41042)
0925 14:53:25.709834 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.709872 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.710326 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.710674 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.710676 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.710933 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.711085 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.711089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.712103 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.712104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.712798 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.712801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.713295 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.713297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.713359 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.713817 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.713825 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.715224 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:25.717322 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.717329 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.717331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.717618 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.717955 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.717956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.717956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.718041 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.718546 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:25.718552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:25.718612 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 14:53:25.718690 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:53:25.718817 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260925 14:53:25.719121 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.709823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.0.98.62:46647)
0925 14:53:25.709937 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.709941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.709975 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:53:25.710555 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.710557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.710574 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.710653 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.711163 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.711999 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.712880 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.713186 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.713267 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.715315 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:25.715322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:25.715322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:25.715328 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:25.717234 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.717722 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.717725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.717726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.717850 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.718109 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.718110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.718111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.718472 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:25.718571 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:25.718753 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 14:53:25.718921 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260925 14:53:25.719916 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:25.720297 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.720164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34732)
0925 14:53:25.720208 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.720268 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:53:25.720515 22609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
I20260925 14:53:25.720959 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.720868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34736)
0925 14:53:25.720918 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.720943 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:53:25.721078 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39757: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 14:53:25.741681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39757
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46647,127.0.98.61:39757,127.0.98.60:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:25.770331 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } attempt: 1
I20260925 14:53:25.770907 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.770759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34746)
0925 14:53:25.770833 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.770876 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:53:25.771247 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.771152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34748)
0925 14:53:25.771196 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.771222 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:53:25.771347 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39757: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
W20260925 14:53:25.824273 22668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:53:25.824534 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:25.824551 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:25.824571 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:25.825740 22668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:25.825785 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:25.825809 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:25.825821 22668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:25.825860 22668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:25.827636 22668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46647,127.0.98.61:39757,127.0.98.60:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:25.827914 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:25.828181 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:25.833303 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:25.833333 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:25.833432 22668 server_base.cc:1034] running on GCE node
W20260925 14:53:25.833303 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:25.833671 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:25.834127 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:25.835296 22668 hybrid_clock.cc:648] HybridClock initialized: now 1790348005835268 us; error 40 us; skew 500 ppm
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:25.837790 22668 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 14:53:25.838150 22668 webserver.cc:460] Webserver started at http://127.0.98.61:42015/ using document root <none> and password file <none>
I20260925 14:53:25.838431 22668 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:25.838508 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:25.838662 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:25.839509 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } attempt: 2
I20260925 14:53:25.839936 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.839784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34764)
0925 14:53:25.839850 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.839905 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:53:25.839923 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data/instance:
uuid: "27633e2ed6a04dba8935a4748b0ddb75"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "97460ae1dd3f3b78e6b27e261f8c8a7d"
server_key_iv: "f81fada23dc15b6d5fb92b0a8124936f"
server_key_version: "encryptionkey@0"
I20260925 14:53:25.840262 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.840165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34776)
0925 14:53:25.840209 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.840233 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:53:25.840375 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal/instance:
uuid: "27633e2ed6a04dba8935a4748b0ddb75"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "97460ae1dd3f3b78e6b27e261f8c8a7d"
server_key_iv: "f81fada23dc15b6d5fb92b0a8124936f"
server_key_version: "encryptionkey@0"
W20260925 14:53:25.840381 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39757: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39757: connect: Connection refused (error 111)
I20260925 14:53:25.841713 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:53:25.842521 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:25.842705 22668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:25.842804 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal
uuid: "27633e2ed6a04dba8935a4748b0ddb75"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "97460ae1dd3f3b78e6b27e261f8c8a7d"
server_key_iv: "f81fada23dc15b6d5fb92b0a8124936f"
server_key_version: "encryptionkey@0"
I20260925 14:53:25.842891 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:25.862107 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:25.864070 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:25.864243 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:25.868338 22668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39757
I20260925 14:53:25.868376 22735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39757 every 8 connection(s)
I20260925 14:53:25.868752 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:53:25.869735 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:25.871366 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:25.876463 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22668
I20260925 14:53:25.876590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-1/wal/instance
I20260925 14:53:25.876830 392 external_mini_cluster.cc:1428] Setting key bd6c20cbf7151152cc98540c35a6a057
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:25.881278 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.872103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41046 (local address 127.0.98.62:46647)
0925 14:53:25.872181 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.872184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.872361 (+ 177us) server_negotiation.cc:407] Connection header received
0925 14:53:25.872864 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.872865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.872881 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.872946 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.873581 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.874086 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.875150 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.875362 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.875463 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.878074 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:25.878075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:25.878075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:25.878082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:25.879747 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.880258 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.880259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.880259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.880310 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.880615 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.880616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.880616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.880743 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:25.880797 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:25.881148 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 14:53:25.881214 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:53:25.881670 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.872084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46647 (local address 127.0.0.1:41046)
0925 14:53:25.872242 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.872277 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.872756 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.873022 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.873025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.873276 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.873469 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.873473 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.874181 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.874182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.875045 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.875047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.875481 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.875482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.875616 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.876139 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.876148 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.877981 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:25.879839 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.879845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.879848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.880181 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.880387 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.880388 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.880389 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.880539 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.880813 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:25.880819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:25.881006 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 14:53:25.881130 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 14:53:25.881309 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260925 14:53:25.882325 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:25.891830 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.882627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34782)
0925 14:53:25.882681 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.882693 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.882738 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.883049 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.883050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.883159 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.883205 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.883206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.884583 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.884584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.885085 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.885087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.885702 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.885703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.885758 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.886112 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.886116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.888507 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:25.890407 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.890408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.890408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.890670 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.890949 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.890949 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.890950 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.891007 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.891293 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:25.891294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:25.891331 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:53:25.891409 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:53:25.891516 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:53:25.892009 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.882708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34782 (local address 127.0.98.61:39757)
0925 14:53:25.882882 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.882887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.882910 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:53:25.882933 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.882935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.882951 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.883032 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.883287 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.884489 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.885170 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.885634 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.885695 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.888613 (+ 2918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:25.888621 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:25.888621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:25.888628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:25.890257 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.890796 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.890800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.890800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.890861 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.891079 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.891080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.891081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.891219 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:25.891314 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:25.891440 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:53:25.891570 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260925 14:53:25.892584 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:25.892972 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.892819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48394)
0925 14:53:25.892905 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.892952 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 14:53:25.893148 22687 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
I20260925 14:53:25.893646 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.893579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48396)
0925 14:53:25.893606 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.893630 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 14:53:25.893782 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 14:53:25.905043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41687
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46647,127.0.98.61:39757,127.0.98.60:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:25.931589 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } attempt: 3
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:25.938943 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.932070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34790 (local address 127.0.98.61:39757)
0925 14:53:25.932140 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:25.932142 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:25.932177 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:53:25.932202 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:25.932203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:25.932213 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:25.932255 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:25.932618 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.933116 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.934135 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.934361 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.934470 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.936476 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:25.936477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:25.936478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:25.936484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:25.937868 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.938226 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.938227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.938227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.938276 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:25.938508 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:25.938509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:25.938510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:25.938611 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:25.938660 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:25.938811 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:53:25.938880 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:53:25.938993 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.931980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34790)
0925 14:53:25.932057 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.932070 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:25.932136 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:25.932278 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:25.932279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:25.932458 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:25.932534 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.932536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.933198 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.933199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.934037 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:25.934041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:25.934466 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:25.934468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:25.934549 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:25.934999 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:25.935005 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:25.936388 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:25.937951 (+ 1563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.937952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.937952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.938155 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.938354 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:25.938354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:25.938355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:25.938442 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:25.938668 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:25.938668 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:25.938715 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:53:25.938785 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:53:25.938874 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 14:53:25.939527 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:25.939859 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.939748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48404)
0925 14:53:25.939801 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.939836 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:53:25.940163 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.940085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48414)
0925 14:53:25.940125 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.940148 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:53:25.940259 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
I20260925 14:53:25.943008 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } attempt: 1
I20260925 14:53:25.943420 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.943277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48416)
0925 14:53:25.943327 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.943397 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 14:53:25.943727 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.943622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48426)
0925 14:53:25.943682 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.943705 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 14:53:25.943817 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
I20260925 14:53:25.971408 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } attempt: 1
I20260925 14:53:25.971961 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.971811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48436)
0925 14:53:25.971884 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.971929 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:53:25.972343 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:25.972250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48444)
0925 14:53:25.972296 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:25.972326 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:53:25.972462 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
W20260925 14:53:25.988881 22743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:53:25.989089 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:25.989106 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:25.989128 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:25.990166 22743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:25.990200 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:25.990214 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:25.990227 22743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:25.990239 22743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:25.991673 22743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46647,127.0.98.61:39757,127.0.98.60:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:25.991859 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:25.992022 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:25.996634 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:25.996641 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:25.996641 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:25.996791 22743 server_base.cc:1034] running on GCE node
I20260925 14:53:25.997116 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:25.997385 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:25.998545 22743 hybrid_clock.cc:648] HybridClock initialized: now 1790348005998537 us; error 39 us; skew 500 ppm
Sep 25 14:53:25 dist-test-slave-z2s5 krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:26.000912 22743 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 14:53:26.001307 22743 webserver.cc:460] Webserver started at http://127.0.98.60:44889/ using document root <none> and password file <none>
I20260925 14:53:26.001521 22743 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:26.001585 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:26.001725 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:26.002851 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data/instance:
uuid: "cb4ae53b4d024b358049afeb21082cfa"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "d8b1dde5d77091745f71270916b9d4fc"
server_key_iv: "fcdd9fc2954620dd64f8bcf5aeb9e7c3"
server_key_version: "encryptionkey@0"
I20260925 14:53:26.003193 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal/instance:
uuid: "cb4ae53b4d024b358049afeb21082cfa"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "d8b1dde5d77091745f71270916b9d4fc"
server_key_iv: "fcdd9fc2954620dd64f8bcf5aeb9e7c3"
server_key_version: "encryptionkey@0"
I20260925 14:53:26.004523 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:26.005282 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:26.005550 22743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:53:26.005666 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal
uuid: "cb4ae53b4d024b358049afeb21082cfa"
format_stamp: "Formatted at 2026-09-25 14:53:25 on dist-test-slave-z2s5"
server_key: "d8b1dde5d77091745f71270916b9d4fc"
server_key_iv: "fcdd9fc2954620dd64f8bcf5aeb9e7c3"
server_key_version: "encryptionkey@0"
I20260925 14:53:26.005739 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:26.011972 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } attempt: 2
I20260925 14:53:26.012440 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.012271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48446)
0925 14:53:26.012331 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.012406 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:53:26.012769 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.012678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48456)
0925 14:53:26.012721 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.012746 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:53:26.012862 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41687: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41687: connect: Connection refused (error 111)
I20260925 14:53:26.024893 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:26.027063 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:26.027220 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:26.031302 22743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41687
I20260925 14:53:26.031402 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41687 every 8 connection(s)
I20260925 14:53:26.031713 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:53:26.032680 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:26.034165 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:26.039873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22743
I20260925 14:53:26.039964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790347739045519-392-0/minicluster-data/master-2/wal/instance
I20260925 14:53:26.040196 392 external_mini_cluster.cc:1428] Setting key f29bf7cffd5abb5e755b0d233c93fed6
I20260925 14:53:26.043515 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.034849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46647 (local address 127.0.0.1:41050)
0925 14:53:26.035011 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.035045 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:26.035517 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:26.035718 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:26.035721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:26.035975 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:26.036160 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.036164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.036765 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.036766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.037575 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.037577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.037907 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.037908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.038009 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.038592 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:26.038600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:26.040032 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:26.042101 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.042105 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.042107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.042408 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.042607 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.042608 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.042608 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.042737 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.042965 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:26.042969 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:26.043107 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:53:26.043217 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:53:26.043364 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 14:53:26.043772 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.034923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.0.98.62:46647)
0925 14:53:26.035002 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.035004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.035127 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:53:26.035602 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.035603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.035612 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.035665 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.036228 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.036679 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.037648 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.037827 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.037882 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.040134 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.040135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.040136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.040142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.042030 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.042488 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.042489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.042489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.042543 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.042800 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.042801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.042801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.042902 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.042955 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.043600 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.043680 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:53:26.044261 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:26.047681 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } attempt: 2
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:26.051848 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.040617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48464 (local address 127.0.98.60:41687)
0925 14:53:26.041124 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.041128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.041150 (+ 22us) server_negotiation.cc:407] Connection header received
0925 14:53:26.041171 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.041173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.041187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.041264 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.042087 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.043187 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.044214 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.044587 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.044662 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.047446 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.047452 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.047453 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.047475 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.049677 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.050100 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.050103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.050103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.050820 (+ 717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.051048 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.051049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.051049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.051242 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.051383 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.051661 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.051734 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":71,"threads_started":1}
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:26.052434 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.044534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39757 (local address 127.0.0.1:34792)
0925 14:53:26.044587 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.044599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:26.044635 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:26.044777 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:26.044777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:26.044883 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:26.044921 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.044922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.045539 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.045540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.046011 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.046012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.046477 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.046478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.046544 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.046937 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:26.046942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:26.048485 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:26.051027 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.051028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.051029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.051236 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.051462 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.051463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.051463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.051542 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.052136 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:26.052137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:26.052181 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 14:53:26.052246 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:53:26.052327 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:53:26.053083 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.044589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.0.98.61:39757)
0925 14:53:26.044631 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.044634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.044644 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:26.044692 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.044693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.044705 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.044768 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.045018 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.045455 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.046092 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.046369 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.046476 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.048587 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.048591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.048591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.048603 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.050680 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.051318 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.051319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.051320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.051379 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.051629 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.051630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.051631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.051758 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.051822 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.052952 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.053024 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:26.053498 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:26.055981 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.047956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48470)
0925 14:53:26.048044 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.048053 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:26.048085 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:26.048347 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:26.048348 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:26.048502 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:26.048564 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.048566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.049286 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.049288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.050160 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.050163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.050610 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.050611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.050667 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.051011 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:26.051016 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:26.052396 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:26.054920 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.054921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.054921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.055135 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.055337 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.055338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.055338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.055404 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.055668 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:26.055669 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:26.055710 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:53:26.055772 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:53:26.055866 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:53:26.056425 22816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.048047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.98.60:41687)
0925 14:53:26.048202 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.048204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.048211 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:53:26.048229 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.048230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.048239 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.048338 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.048652 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.049182 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.050250 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.050529 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.050609 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.052461 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.052461 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.052462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.052468 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.054842 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.055205 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.055206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.055206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.055265 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.055475 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.055476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.055476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.055613 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.055675 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.056177 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.056258 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:26.058063 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f: Bootstrap starting.
I20260925 14:53:26.058918 22661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:26.059248 22661 log.cc:824] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:26.060194 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f: No bootstrap required, opened a new log
I20260925 14:53:26.061241 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.053718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48472)
0925 14:53:26.054228 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.054237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:26.054283 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:26.054457 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:26.054457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:26.054583 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:26.054619 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.054621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.055375 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.055376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.056001 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.056003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.056454 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.056454 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.056502 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.056880 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:26.056885 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:26.058243 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:26.060067 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.060068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.060068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.060277 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.060575 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.060576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.060576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.060622 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.060890 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:26.060890 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:26.060928 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:53:26.060995 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 14:53:26.061086 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496}
I20260925 14:53:26.061285 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.054190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.0.98.60:41687)
0925 14:53:26.054243 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.054245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.054279 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:53:26.054344 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.054345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.054355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.054447 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.054698 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.055311 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.056112 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.056381 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.056455 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.058342 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.058343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.058344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.058350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.059971 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.060464 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.060465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.060466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.060512 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.060682 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.060683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.060683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.060783 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.060869 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.061018 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.061133 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":34}
I20260925 14:53:26.061820 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.052712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.0.98.62:46647)
0925 14:53:26.052756 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.052759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.052768 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:26.052786 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.052787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.052795 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.052843 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.053542 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.054078 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.054978 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.055279 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.055354 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.058483 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.058484 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.058484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.058490 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.060227 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.060702 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.060703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.060704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.060762 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.060966 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.060967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.060967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.061097 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.061230 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.061656 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.061731 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:53:26.062505 22661 raft_consensus.cc:348] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.062662 22661 raft_consensus.cc:729] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e61c305f741c43c780ebc9a23df79d7f, State: Initialized, Role: FOLLOWER
I20260925 14:53:26.062785 22661 consensus_queue.cc:260] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.062956 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: Bootstrap starting.
I20260925 14:53:26.063082 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.063162 22661 sys_catalog.cc:564] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:26.063166 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:26.063575 22811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:26.063879 22811 log.cc:824] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:26.064579 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: No bootstrap required, opened a new log
I20260925 14:53:26.066721 22811 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.066898 22811 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb4ae53b4d024b358049afeb21082cfa, State: Initialized, Role: FOLLOWER
I20260925 14:53:26.066988 22811 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.067284 22811 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:26.066176 22593 master_runner.cc:186] Error getting master registration for 127.0.98.61:39757: OK, instance_id { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" instance_seqno: 1790348005867160 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:53:26.069173 22593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 14:53:26.064864 22830 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:26.069294 22830 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:26.069447 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.069787 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:53:26.070802 22843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:26.070958 22843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:26.072218 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.062380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34796 (local address 127.0.98.61:39757)
0925 14:53:26.062483 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.062485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.062496 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:26.062574 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.062575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.062584 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.062626 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.063257 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.064015 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.065066 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.065378 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.065438 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.068250 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.068251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.068251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.068259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.070555 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.071096 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.071098 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.071098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.071160 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.071350 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.071351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.071352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.071475 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.071623 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.072043 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.072119 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 14:53:26.070979 22743 master_runner.cc:186] Error getting master registration for 127.0.98.61:39757: OK, instance_id { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" instance_seqno: 1790348005867160 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:53:26.072634 22743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:53:26.073477 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:53:26.074280 22635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e61c305f741c43c780ebc9a23df79d7f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41056
I20260925 14:53:26.074442 22635 raft_consensus.cc:482] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:53:26.074503 22635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:26.075129 22635 raft_consensus.cc:504] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.075326 22635 leader_election.cc:288] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [CANDIDATE]: Term 1 election: Requested vote from peers 27633e2ed6a04dba8935a4748b0ddb75 (127.0.98.61:39757), cb4ae53b4d024b358049afeb21082cfa (127.0.98.60:41687)
I20260925 14:53:26.075657 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e61c305f741c43c780ebc9a23df79d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb4ae53b4d024b358049afeb21082cfa"
I20260925 14:53:26.075762 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:26.076610 22785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61c305f741c43c780ebc9a23df79d7f in term 1.
I20260925 14:53:26.076792 22612 leader_election.cc:302] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb4ae53b4d024b358049afeb21082cfa, e61c305f741c43c780ebc9a23df79d7f; no voters:
I20260925 14:53:26.077026 22818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:26.077199 22818 raft_consensus.cc:686] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [term 1 LEADER]: Becoming Leader. State: Replica: e61c305f741c43c780ebc9a23df79d7f, State: Running, Role: LEADER
I20260925 14:53:26.077309 22818 consensus_queue.cc:237] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.077744 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61c305f741c43c780ebc9a23df79d7f. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.077831 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:26.082572 22851 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:26.082782 22851 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:26.085072 22785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [term 1 FOLLOWER]: Refusing update from remote peer e61c305f741c43c780ebc9a23df79d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:26.085425 22845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:26.086800 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61c305f741c43c780ebc9a23df79d7f. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.086964 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:26.096040 22852 mvcc.cc:205] Tried to move back new op lower bound from 7333265432920047616 to 7333265432893677568. Current Snapshot: MvccSnapshot[applied={T|T < 7333265432920047616}]
I20260925 14:53:26.096567 22818 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.096632 22740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.083799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34800 (local address 127.0.98.61:39757)
0925 14:53:26.083842 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.083845 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.084613 (+ 768us) server_negotiation.cc:407] Connection header received
0925 14:53:26.084696 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.084697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.084708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.084775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.085966 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.086778 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.088330 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.088638 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.088680 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.091318 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.091319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.091319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.091338 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.093110 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.095020 (+ 1910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.095022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.095023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.095094 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.095655 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.095656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.095657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.095805 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.095890 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.096453 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.096536 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:26.096654 22818 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:26.097517 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.097601 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:26.097787 22856 mvcc.cc:205] Tried to move back new op lower bound from 7333265432920047616 to 7333265432893677568. Current Snapshot: MvccSnapshot[applied={T|T < 7333265432920047616}]
I20260925 14:53:26.102558 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.084325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48480 (local address 127.0.98.60:41687)
0925 14:53:26.084379 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.084381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.090583 (+ 6202us) server_negotiation.cc:407] Connection header received
0925 14:53:26.090739 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.090740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.090761 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.090791 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.091748 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.092485 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.094014 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.094326 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.094365 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.099247 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.099248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.099260 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.099267 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.100883 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.101489 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.101491 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.101491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.101557 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.101838 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.101839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.101840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.101978 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.102049 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.102361 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.102467 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:53:26.103964 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } attempt: 3
I20260925 14:53:26.106073 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.084064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.0.98.62:46647)
0925 14:53:26.084102 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.084104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.084783 (+ 679us) server_negotiation.cc:407] Connection header received
0925 14:53:26.084898 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.084899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.084910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.084943 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.094078 (+ 9135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.094779 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.097775 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.098095 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.098136 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.101453 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.101454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.101455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.101462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.103270 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.103972 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.103973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.103974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.104052 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.104624 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.104625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.104625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.104767 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.104846 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.105890 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.105973 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:26.106626 22851 catalog_manager.cc:1347] Generated new cluster ID: 04ef0e7c50b54f759abc5f9c9b97a6d9
I20260925 14:53:26.106673 22851 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:53:26 dist-test-slave-z2s5 krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348005, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:26.118865 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.104215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41687 (local address 127.0.0.1:48488)
0925 14:53:26.104358 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:26.104372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:26.104421 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:26.104686 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:26.104687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:26.104861 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:26.104903 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.104905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.105823 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.105824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.106648 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:26.106650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.107193 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.107193 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:26.107263 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.107662 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:26.107679 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:26.115456 (+ 7777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:26.117643 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.117644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.117645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.117881 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.118121 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:26.118122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:26.118122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:26.118191 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:26.118543 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:26.118544 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:26.118601 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 14:53:26.118663 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:53:26.118764 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 14:53:26.119251 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:26.104503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.0.98.60:41687)
0925 14:53:26.104548 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.104550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.104556 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:53:26.104585 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.104586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.104595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.104639 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.104992 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.105720 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.106751 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:26.107073 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:26.107132 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:26.115583 (+ 8451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:26.115585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:26.115585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:26.115592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:26.117524 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.117973 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.117974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.117974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.118042 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:26.118264 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:26.118265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:26.118266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:26.118445 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:26.118505 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:26.119083 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 14:53:26.119160 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:53:26.120782 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: Bootstrap starting.
I20260925 14:53:26.121512 22736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:26.121821 22736 log.cc:824] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:26.125789 22851 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:26.127110 22851 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:26.127351 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: No bootstrap required, opened a new log
I20260925 14:53:26.129411 22736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.129563 22736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27633e2ed6a04dba8935a4748b0ddb75, State: Initialized, Role: FOLLOWER
I20260925 14:53:26.129647 22736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } }
I20260925 14:53:26.130008 22736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:26.130100 22709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:26.131178 22709 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [term 1 FOLLOWER]: Refusing update from remote peer e61c305f741c43c780ebc9a23df79d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:26.131517 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e61c305f741c43c780ebc9a23df79d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27633e2ed6a04dba8935a4748b0ddb75"
I20260925 14:53:26.132364 22710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e61c305f741c43c780ebc9a23df79d7f in term 1.
I20260925 14:53:26.132628 22845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:26.138484 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.138578 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:26.138619 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e61c305f741c43c780ebc9a23df79d7f. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.138659 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:26.138684 22862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.138725 22862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:53:26.139447 22873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:26.139518 22873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:26.140440 22851 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f: Generated new TSK 0
I20260925 14:53:26.140674 22851 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:26.142943 22875 mvcc.cc:205] Tried to move back new op lower bound from 7333265433086803968 to 7333265432893677568. Current Snapshot: MvccSnapshot[applied={T|T < 7333265432920047616}]
I20260925 14:53:26.143777 22845 sys_catalog.cc:455] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e61c305f741c43c780ebc9a23df79d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e61c305f741c43c780ebc9a23df79d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46647 } } peers { permanent_uuid: "27633e2ed6a04dba8935a4748b0ddb75" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39757 } } peers { permanent_uuid: "cb4ae53b4d024b358049afeb21082cfa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41687 } } }
I20260925 14:53:26.143885 22845 sys_catalog.cc:458] T 00000000000000000000000000000000 P e61c305f741c43c780ebc9a23df79d7f [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:26.161365 22815 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.159724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.0.98.60:41687)
0925 14:53:26.159786 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.159788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.160634 (+ 846us) server_negotiation.cc:407] Connection header received
0925 14:53:26.160783 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.160784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.160795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.160826 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.161297 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48498: BlockingRecv error: recv got EOF from 127.0.0.1:48498 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:53:26.162216 22664 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.160181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.0.98.62:46647)
0925 14:53:26.160220 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.160222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.161735 (+ 1513us) server_negotiation.cc:407] Connection header received
0925 14:53:26.161827 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.161828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.161839 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.161867 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.162160 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41066: BlockingRecv error: recv got EOF from 127.0.0.1:41066 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:53:26.163404 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.162782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34816 (local address 127.0.98.61:39757)
0925 14:53:26.162818 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.162820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.162923 (+ 103us) server_negotiation.cc:407] Connection header received
0925 14:53:26.163021 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.163021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.163031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.163060 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.163336 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34816: BlockingRecv error: recv got EOF from 127.0.0.1:34816 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:53:26.165719 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.165039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34830 (local address 127.0.98.61:39757)
0925 14:53:26.165071 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.165073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.165243 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:53:26.165348 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.165348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.165365 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.165392 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.165675 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34830: BlockingRecv error: recv got EOF from 127.0.0.1:34830 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:53:26.168859 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.168093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.0.98.61:39757)
0925 14:53:26.168141 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.168143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.168381 (+ 238us) server_negotiation.cc:407] Connection header received
0925 14:53:26.168473 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.168474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.168483 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.168513 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.168815 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34840: BlockingRecv error: recv got EOF from 127.0.0.1:34840 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:53:26.172899 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.172125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.0.98.61:39757)
0925 14:53:26.172161 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.172163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.172385 (+ 222us) server_negotiation.cc:407] Connection header received
0925 14:53:26.172486 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.172487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.172497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.172528 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.172854 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34844: BlockingRecv error: recv got EOF from 127.0.0.1:34844 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:53:26.179174 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.178183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.0.98.61:39757)
0925 14:53:26.178219 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.178221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.178641 (+ 420us) server_negotiation.cc:407] Connection header received
0925 14:53:26.178742 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.178743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.178753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.178782 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.179130 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34860: BlockingRecv error: recv got EOF from 127.0.0.1:34860 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:53:26.189465 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.188569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34868 (local address 127.0.98.61:39757)
0925 14:53:26.188608 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.188610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.188852 (+ 242us) server_negotiation.cc:407] Connection header received
0925 14:53:26.188957 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.188958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.188969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.189015 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.189399 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34868: BlockingRecv error: recv got EOF from 127.0.0.1:34868 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:53:26.208055 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.207100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.98.61:39757)
0925 14:53:26.207146 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.207148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.207400 (+ 252us) server_negotiation.cc:407] Connection header received
0925 14:53:26.207506 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.207507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.207519 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.207550 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.207993 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34874: BlockingRecv error: recv got EOF from 127.0.0.1:34874 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:53:26.242954 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.241711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.0.98.61:39757)
0925 14:53:26.241754 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.241756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.242043 (+ 287us) server_negotiation.cc:407] Connection header received
0925 14:53:26.242179 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.242180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.242192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.242245 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.242895 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34884: BlockingRecv error: recv got EOF from 127.0.0.1:34884 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 14:53:26.309463 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.308538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34894 (local address 127.0.98.61:39757)
0925 14:53:26.308613 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.308616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.308868 (+ 252us) server_negotiation.cc:407] Connection header received
0925 14:53:26.308948 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.308949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.308962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.309031 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.309406 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34894: BlockingRecv error: recv got EOF from 127.0.0.1:34894 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 14:53:26.439630 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.438861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34906 (local address 127.0.98.61:39757)
0925 14:53:26.438934 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.438936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.439051 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:53:26.439100 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.439101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.439113 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.439174 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.439562 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34906: BlockingRecv error: recv got EOF from 127.0.0.1:34906 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 14:53:26.697294 22740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:26.696669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34922 (local address 127.0.98.61:39757)
0925 14:53:26.696711 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:26.696713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:26.696767 (+ 54us) server_negotiation.cc:407] Connection header received
0925 14:53:26.696823 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:26.696823 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:26.696833 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:26.696926 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:26.697203 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34922: BlockingRecv error: recv got EOF from 127.0.0.1:34922 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:27.071405 22843 catalog_manager.cc:1259] Loaded cluster ID: 04ef0e7c50b54f759abc5f9c9b97a6d9
I20260925 14:53:27.071481 22843 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: loading cluster ID for follower catalog manager: success
I20260925 14:53:27.072248 22843 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: acquiring CA information for follower catalog manager: success
I20260925 14:53:27.072513 22843 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb4ae53b4d024b358049afeb21082cfa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:53:27.139969 22873 catalog_manager.cc:1259] Loaded cluster ID: 04ef0e7c50b54f759abc5f9c9b97a6d9
I20260925 14:53:27.140049 22873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: loading cluster ID for follower catalog manager: success
I20260925 14:53:27.141012 22873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: acquiring CA information for follower catalog manager: success
I20260925 14:53:27.141333 22873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27633e2ed6a04dba8935a4748b0ddb75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:53:27.213001 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.210431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.0.98.61:39757)
0925 14:53:27.210554 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.210556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.210570 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:53:27.210649 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.210650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.210700 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.210782 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:27.210942 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.211441 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.212237 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.212436 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.212500 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.212637 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:27.212746 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:27.212803 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.212856 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.212873 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260925 14:53:27.217310 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.214747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.0.98.60:41687)
0925 14:53:27.214845 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.214847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.214856 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:27.214931 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.214932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.214983 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.215068 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:27.215183 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.215656 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.216443 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.216665 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.216736 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.216898 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:27.217064 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:27.217172 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.217196 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.217219 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":48,"threads_started":1}
I20260925 14:53:27.222061 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.219128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.98.61:39757)
0925 14:53:27.219188 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.219191 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.219225 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:53:27.219335 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.219336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.219348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.219380 (+ 32us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:27.219563 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.220105 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.221233 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.221529 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.221564 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.221787 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:27.221838 (+ 51us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:27.221887 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.221978 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.221998 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:27.222039 22950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.219024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48506 (local address 127.0.98.60:41687)
0925 14:53:27.219062 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.219064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.219264 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:53:27.219329 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.219330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.219341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.219393 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:27.219507 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.220221 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.221233 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.221522 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.221563 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.221754 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:27.221784 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:27.221839 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.221942 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.221968 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:27.222075 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:27.219016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41078 (local address 127.0.98.62:46647)
0925 14:53:27.219134 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:27.219137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:27.219216 (+ 79us) server_negotiation.cc:407] Connection header received
0925 14:53:27.219454 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:27.219455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:27.219506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:27.219579 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:27.219747 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.220452 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.221229 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:27.221454 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:27.221524 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:27.221722 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:27.221834 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:27.221892 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:27.221943 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:53:27.221965 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260925 14:53:27.224002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22593
I20260925 14:53:27.229336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22668
I20260925 14:53:27.234637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22743
2026-09-25T14:53:27Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2806 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:53:27 dist-test-slave-z2s5 krb5kdc[22961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:53:27 dist-test-slave-z2s5 krb5kdc[22961](info): set up 2 sockets
Sep 25 14:53:27 dist-test-slave-z2s5 krb5kdc[22961](info): commencing operation
krb5kdc: starting...
W20260925 14:53:28.274718 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/test-user.keytab.
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260925 14:53:28.398100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46145,127.0.98.61:37193,127.0.98.60:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:53:28.487341 22974 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:53:28.487684 22974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:28.487744 22974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:28.487788 22974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:28.489575 22974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:28.489661 22974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:28.489687 22974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:28.489706 22974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:28.489733 22974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:28.492368 22974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46145,127.0.98.61:37193,127.0.98.60:40125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:28.492678 22974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:28.492966 22974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:28.498503 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.498672 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.498581 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:28.498631 22974 server_base.cc:1034] running on GCE node
I20260925 14:53:28.499017 22974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:28.499390 22974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:28.500550 22974 hybrid_clock.cc:648] HybridClock initialized: now 1790348008500523 us; error 41 us; skew 500 ppm
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:28.503345 22974 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260925 14:53:28.503963 22974 webserver.cc:460] Webserver started at http://127.0.98.62:43251/ using document root <none> and password file <none>
I20260925 14:53:28.504261 22974 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:28.504324 22974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:28.504464 22974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:28.506096 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data/instance:
uuid: "6d07220ad6094547ade597171f1c8588"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "4753c354697a9ad54f9836e2ef19f033"
server_key_iv: "bc70cf628cfb876a07474ad4a1dddd83"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.506592 22974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal/instance:
uuid: "6d07220ad6094547ade597171f1c8588"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "4753c354697a9ad54f9836e2ef19f033"
server_key_iv: "bc70cf628cfb876a07474ad4a1dddd83"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.508328 22974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:53:28.509177 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:28.509341 22974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:28.509457 22974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal
uuid: "6d07220ad6094547ade597171f1c8588"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "4753c354697a9ad54f9836e2ef19f033"
server_key_iv: "bc70cf628cfb876a07474ad4a1dddd83"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.509531 22974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:28.535475 22974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:28.537374 22974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:28.537540 22974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:28.541714 22974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46145
I20260925 14:53:28.541751 23041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46145 every 8 connection(s)
I20260925 14:53:28.542105 22974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/data/info.pb
I20260925 14:53:28.543133 23042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:28.544286 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 22974
I20260925 14:53:28.544381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-0/wal/instance
I20260925 14:53:28.544584 392 external_mini_cluster.cc:1428] Setting key 6d79e97e4350b0ff65b21cc8c533da19
I20260925 14:53:28.545125 23042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260925 14:53:28.556159 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.546000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42736 (local address 127.0.98.62:46145)
0925 14:53:28.546176 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.546186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.546414 (+ 228us) server_negotiation.cc:407] Connection header received
0925 14:53:28.546964 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.546967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.546990 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.547115 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.547853 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.548781 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.549691 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.550236 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.550352 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.552507 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.552519 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.552519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.552525 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.554371 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.554938 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.554942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.554942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.555043 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.555268 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.555269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.555269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.555493 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.555589 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.555771 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.555910 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"spinlock_wait_cycles":8576,"thread_start_us":89,"threads_started":1}
I20260925 14:53:28.556161 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.545985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46145 (local address 127.0.0.1:42736)
0925 14:53:28.546257 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.546284 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.546805 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.547128 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.547132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.547517 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.547754 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.547764 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.548917 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.548919 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.549605 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.549607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.550342 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.550343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.550444 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.550996 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.551004 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.552420 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.554584 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.554590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.554592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.554847 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.555106 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.555107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.555107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.555189 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.555567 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.555572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.555634 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.555746 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.555972 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":86,"thread_start_us":86,"threads_started":1}
I20260925 14:53:28.557186 23042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:28.557545 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.557425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33224)
0925 14:53:28.557473 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.557516 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 14:53:28.557704 22991 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
I20260925 14:53:28.558117 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.558030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33238)
0925 14:53:28.558078 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.558101 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 14:53:28.558238 23042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260925 14:53:28.569237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37193
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46145,127.0.98.61:37193,127.0.98.60:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:28.607453 23042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } attempt: 1
I20260925 14:53:28.607985 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.607834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33246)
0925 14:53:28.607901 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.607941 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 14:53:28.608443 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.608309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33250)
0925 14:53:28.608369 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.608413 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:53:28.608577 23042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
W20260925 14:53:28.650116 23049 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:53:28.650345 23049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:28.650363 23049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:28.650439 23049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:28.651572 23049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:28.651613 23049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:28.651628 23049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:28.651640 23049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:28.651697 23049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:28.653256 23049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46145,127.0.98.61:37193,127.0.98.60:40125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:28.653542 23049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:28.653774 23049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:53:28.658631 23054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.658624 23055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.658623 23057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:28.658862 23049 server_base.cc:1034] running on GCE node
I20260925 14:53:28.659142 23049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:28.659447 23049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:28.660671 23049 hybrid_clock.cc:648] HybridClock initialized: now 1790348008660658 us; error 45 us; skew 500 ppm
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:28.663245 23049 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260925 14:53:28.663739 23049 webserver.cc:460] Webserver started at http://127.0.98.61:36429/ using document root <none> and password file <none>
I20260925 14:53:28.663962 23049 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:28.664028 23049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:28.664182 23049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:28.665226 23049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data/instance:
uuid: "09bc78cc24a646a1b5bbfac4fcca79e9"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "90455f26a12d395617604ca98cdf6cd9"
server_key_iv: "f8323727655997797189d8da8844b132"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.665582 23049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal/instance:
uuid: "09bc78cc24a646a1b5bbfac4fcca79e9"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "90455f26a12d395617604ca98cdf6cd9"
server_key_iv: "f8323727655997797189d8da8844b132"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.667037 23049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:53:28.667753 23064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:28.667917 23049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:53:28.668020 23049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal
uuid: "09bc78cc24a646a1b5bbfac4fcca79e9"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "90455f26a12d395617604ca98cdf6cd9"
server_key_iv: "f8323727655997797189d8da8844b132"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.668092 23049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:28.676750 23042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } attempt: 2
I20260925 14:53:28.677179 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.677032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33260)
0925 14:53:28.677096 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.677141 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:53:28.677544 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.677451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33276)
0925 14:53:28.677497 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.677524 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 14:53:28.677681 23042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:37193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:37193: connect: Connection refused (error 111)
I20260925 14:53:28.686357 23049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:28.688295 23049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:28.688453 23049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:28.692415 23049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:37193
I20260925 14:53:28.692446 23116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:37193 every 8 connection(s)
I20260925 14:53:28.692795 23049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/data/info.pb
I20260925 14:53:28.693783 23117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:28.693959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 23049
I20260925 14:53:28.694031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-1/wal/instance
I20260925 14:53:28.694219 392 external_mini_cluster.cc:1428] Setting key ba6f750c8b07137c3d4a6683a6f546f3
I20260925 14:53:28.695560 23117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:28.708750 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.696399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.0.98.62:46145)
0925 14:53:28.696469 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.696472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.696672 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:53:28.697182 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.697184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.697199 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.697286 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.697901 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.698532 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.700230 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.700530 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.700617 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.705526 (+ 4909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.705528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.705528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.705536 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.707327 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.707719 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.707720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.707721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.707786 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.708030 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.708031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.708032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.708200 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.708281 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.708542 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.708635 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:53:28.708787 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.696398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46145 (local address 127.0.0.1:42750)
0925 14:53:28.696557 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.696592 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.697070 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.697364 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.697367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.697611 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.697805 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.697809 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.698661 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.698662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.700081 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.700084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.700647 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.700648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.700806 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.701405 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.701416 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.705425 (+ 4009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.707413 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.707419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.707421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.707641 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.707855 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.707856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.707856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.707954 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.708282 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.708287 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.708407 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.708495 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.708616 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260925 14:53:28.709407 23117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:28.717599 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.709659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33282)
0925 14:53:28.709705 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.709731 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.709774 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.710025 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.710026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.710162 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.710211 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.710218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.711128 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.711129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.711633 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.711634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.712125 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.712125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.712188 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.712604 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.712609 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.714774 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.716548 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.716549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.716549 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.716712 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.716960 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.716961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.716961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.717019 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.717324 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.717325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.717376 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.717453 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.717534 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:53:28.717710 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.709755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33282 (local address 127.0.98.61:37193)
0925 14:53:28.709896 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.709900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.709916 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:53:28.709930 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.709931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.709940 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.710012 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.710289 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.711033 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.711732 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.712035 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.712125 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.714877 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.714886 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.714886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.714893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.716467 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.716824 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.716827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.716828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.716893 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.717074 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.717075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.717075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.717250 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.717320 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.717491 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.717582 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":120,"thread_start_us":98,"threads_started":1}
I20260925 14:53:28.718338 23117 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:28.718739 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.718607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44888)
0925 14:53:28.718674 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.718720 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:53:28.718897 23067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
I20260925 14:53:28.719308 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.719245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44892)
0925 14:53:28.719269 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.719292 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 14:53:28.719439 23117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260925 14:53:28.721302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskM77lMC/build/release/bin/kudu
/tmp/dist-test-taskM77lMC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40125
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46145,127.0.98.61:37193,127.0.98.60:40125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:53:28.768666 23117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } attempt: 1
I20260925 14:53:28.768872 23042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } attempt: 3
I20260925 14:53:28.769258 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.769064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44902)
0925 14:53:28.769138 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.769189 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:53:28.769660 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.769527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44906)
0925 14:53:28.769595 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.769626 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 14:53:28.769798 23117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:28.776736 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.769188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33288)
0925 14:53:28.769246 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.769257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.769297 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.769479 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.769480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.769770 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.769855 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.769859 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.770635 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.770636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.771502 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.771505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.771890 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.771891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.771990 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.772459 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.772465 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.773918 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.775647 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.775648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.775649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.775822 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.776063 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.776064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.776064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.776126 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.776369 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.776370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.776416 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.776537 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.776637 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 14:53:28.776731 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.769255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.0.98.61:37193)
0925 14:53:28.769295 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.769297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.769309 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:53:28.769374 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.769375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.769387 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.769432 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.769962 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.770550 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.771589 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.771813 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.771870 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.774059 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.774061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.774061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.774067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.775539 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.775914 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.775915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.775915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.775970 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.776189 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.776190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.776190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.776302 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.776376 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.776566 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.776639 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:53:28.777118 23042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:28.777465 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.777366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44922)
0925 14:53:28.777413 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.777439 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 14:53:28.777793 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.777699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44926)
0925 14:53:28.777749 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.777771 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 14:53:28.777912 23042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
W20260925 14:53:28.803684 23124 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 14:53:28.803910 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:53:28.803928 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:53:28.803949 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:53:28.805119 23124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:53:28.805199 23124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:53:28.805228 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:53:28.805263 23124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:53:28.805284 23124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:53:28.806792 23124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46145,127.0.98.61:37193,127.0.98.60:40125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:53:28.806999 23124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:53:28.807178 23124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:53:28.809067 23042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } attempt: 1
I20260925 14:53:28.809532 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.809384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44942)
0925 14:53:28.809455 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.809501 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:53:28.809900 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.809816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44954)
0925 14:53:28.809856 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.809882 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 14:53:28.810021 23042 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
W20260925 14:53:28.812355 23130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.812383 23129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:53:28.812402 23132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:53:28.812516 23124 server_base.cc:1034] running on GCE node
I20260925 14:53:28.812892 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:53:28.813164 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:53:28.814323 23124 hybrid_clock.cc:648] HybridClock initialized: now 1790348008814308 us; error 29 us; skew 500 ppm
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:53:28.816867 23124 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260925 14:53:28.817406 23124 webserver.cc:460] Webserver started at http://127.0.98.60:39291/ using document root <none> and password file <none>
I20260925 14:53:28.817679 23124 fs_manager.cc:359] Metadata directory not provided
I20260925 14:53:28.817728 23124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:53:28.817900 23124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:53:28.819082 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data/instance:
uuid: "a94b6dd0b97d47b6a7632d800467ce93"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "1b0acf2da5ed094ab0483b56bf192330"
server_key_iv: "8f5b3a9efd2c282a4be21b02f11d6aee"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.819438 23124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal/instance:
uuid: "a94b6dd0b97d47b6a7632d800467ce93"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "1b0acf2da5ed094ab0483b56bf192330"
server_key_iv: "8f5b3a9efd2c282a4be21b02f11d6aee"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.820720 23124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:53:28.821491 23139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:53:28.821689 23124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:53:28.821784 23124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal
uuid: "a94b6dd0b97d47b6a7632d800467ce93"
format_stamp: "Formatted at 2026-09-25 14:53:28 on dist-test-slave-z2s5"
server_key: "1b0acf2da5ed094ab0483b56bf192330"
server_key_iv: "8f5b3a9efd2c282a4be21b02f11d6aee"
server_key_version: "encryptionkey@0"
I20260925 14:53:28.821859 23124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:53:28.837967 23117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } attempt: 2
I20260925 14:53:28.838495 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.838318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44964)
0925 14:53:28.838381 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.838464 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:53:28.838936 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.838813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:44972)
0925 14:53:28.838868 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.838907 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 14:53:28.839068 23117 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40125: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40125: connect: Connection refused (error 111)
I20260925 14:53:28.842795 23124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:53:28.844707 23124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:53:28.844869 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:53:28.848934 23124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40125
I20260925 14:53:28.848996 23191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40125 every 8 connection(s)
I20260925 14:53:28.849346 23124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/data/info.pb
I20260925 14:53:28.850318 23192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:53:28.851869 23192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:53:28.855939 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskM77lMC/build/release/bin/kudu as pid 23124
I20260925 14:53:28.856050 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskM77lMC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790347739045519-392-0/minicluster-data/master-2/wal/instance
I20260925 14:53:28.856268 392 external_mini_cluster.cc:1428] Setting key 3120e5078fc723609a62117c9533091a
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260925 14:53:28.861013 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.852564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42766 (local address 127.0.98.62:46145)
0925 14:53:28.852628 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.852630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.852869 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:53:28.853297 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.853299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.853308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.853355 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.853927 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.854430 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.855506 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.855685 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.855738 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.857979 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.857981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.857981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.857989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.859605 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.860026 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.860026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.860027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.860081 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.860368 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.860369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.860369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.860491 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.860545 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.860885 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.860950 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:53:28.861260 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.852552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46145 (local address 127.0.0.1:42766)
0925 14:53:28.852754 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.852788 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.853219 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.853417 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.853420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.853656 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.853856 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.853860 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.854514 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.854515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.855435 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.855438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.855764 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.855765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.855866 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.856487 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.856494 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.857887 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.859681 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.859687 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.859689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.859951 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.860147 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.860148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.860148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.860271 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.860560 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.860566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.860738 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.860867 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.861035 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":109,"threads_started":1}
I20260925 14:53:28.861887 23192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:28.865553 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.856675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.0.98.60:40125)
0925 14:53:28.856828 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.856832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.856855 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:53:28.856916 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.856918 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.856938 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.857010 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.857605 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.858436 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.859554 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.859841 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.859900 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.862573 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.862580 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.862581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.862587 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.864402 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.864722 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.864725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.864725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.864788 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.864990 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.864991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.864991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.865137 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.865294 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.865367 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.865444 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:28.868673 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.862276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33294 (local address 127.0.98.61:37193)
0925 14:53:28.862323 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.862325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.862334 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:53:28.862357 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.862358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.862367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.862454 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.862743 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.863173 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.864093 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.864308 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.864362 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.866134 (+ 1772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.866135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.866135 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.866142 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.867580 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.867924 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.867925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.867925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.867977 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.868180 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.868181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.868181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.868292 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.868361 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.868516 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.868585 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:53:28.868675 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.862189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:37193 (local address 127.0.0.1:33294)
0925 14:53:28.862246 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.862257 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.862298 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.862458 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.862458 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.862607 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.862662 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.862663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.863249 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.863250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.864018 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.864021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.864380 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.864381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.864459 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.864858 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.864863 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.866043 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.867679 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.867680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.867680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.867840 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.868051 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.868052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.868052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.868106 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.868367 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.868368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.868406 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.868490 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.868578 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 14:53:28.869045 23192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:28.873906 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.866427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.0.98.62:46145)
0925 14:53:28.866639 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.866640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.866647 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:53:28.866664 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.866664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.866672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.866769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.867343 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.867787 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.868478 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.868678 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.868754 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.871264 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.871265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.871266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.871272 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.872719 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.873142 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.873143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.873144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.873204 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.873433 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.873433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.873434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.873524 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.873706 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.873743 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.873811 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260925 14:53:28.876108 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.869303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:45002)
0925 14:53:28.869344 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.869353 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.869385 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.869516 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.869516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.869617 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.869644 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.869645 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.870562 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.870563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.871089 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.871090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.871539 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.871539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.871588 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.871917 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.871922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.873174 (+ 1252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.875147 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.875147 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.875148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.875303 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.875501 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.875502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.875503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.875553 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.875807 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.875808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.875844 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.875912 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.876005 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:53:28.876159 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.869354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.0.98.60:40125)
0925 14:53:28.869391 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.869393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.869426 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:53:28.869445 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.869446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.869456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.869515 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.869710 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.870482 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.871178 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.871482 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.871542 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.873276 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.873278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.873278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.873284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.875053 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.875383 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.875384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.875384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.875431 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.875639 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.875640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.875640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.875743 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.875815 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.875937 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.876049 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":21}
I20260925 14:53:28.877683 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: Bootstrap starting.
I20260925 14:53:28.878578 23192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: Neither blocks nor log segments found. Creating new log.
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260925 14:53:28.878917 23192 log.cc:824] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:28.879807 23192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: No bootstrap required, opened a new log
I20260925 14:53:28.881763 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.874592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33298 (local address 127.0.98.61:37193)
0925 14:53:28.874642 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.874644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.874652 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:53:28.874675 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.874676 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.874685 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.874755 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.875169 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.875611 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.876541 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.876814 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.876893 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.879151 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.879152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.879153 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.879160 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.880661 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.881040 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.881041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.881041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.881095 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.881304 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.881304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.881305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.881413 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.881547 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.881644 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.881703 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:53:28.881842 23192 raft_consensus.cc:348] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.882009 23192 raft_consensus.cc:729] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94b6dd0b97d47b6a7632d800467ce93, State: Initialized, Role: FOLLOWER
I20260925 14:53:28.882097 23192 consensus_queue.cc:260] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.882495 23192 sys_catalog.cc:564] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:28.882445 23198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.882592 23198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:28.883486 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 14:53:28.884274 23209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:28.884331 23209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:28.885180 23042 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } attempt: 2
I20260925 14:53:28.884714 23016 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6d07220ad6094547ade597171f1c8588"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42774
I20260925 14:53:28.886111 23124 master_runner.cc:186] Error getting master registration for 127.0.98.62:46145: OK, instance_id { permanent_uuid: "6d07220ad6094547ade597171f1c8588" instance_seqno: 1790348008540547 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:53:28.889058 23124 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:28.892709 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.885479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.0.98.60:40125)
0925 14:53:28.885517 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.885518 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.885558 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:53:28.885639 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.885640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.885649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.885703 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.886019 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.886784 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.887808 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.888078 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.889220 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.890256 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.890257 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.890257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.890262 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.891660 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.892041 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.892042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.892042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.892088 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.892277 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.892277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.892277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.892387 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.892436 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.892593 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.892655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:53:28.892776 23045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.885431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:45016)
0925 14:53:28.885517 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.885528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.885573 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.885712 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.885713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.885877 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.885913 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.885915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.886888 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.886890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.887710 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.887712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.888165 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.888166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.888224 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.888681 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.888686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.890157 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.891731 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.891732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.891732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.891963 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.892145 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.892146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.892146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.892212 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.892437 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.892438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.892492 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.892579 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.892680 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 14:53:28.894446 23042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588: Bootstrap starting.
I20260925 14:53:28.895129 23042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:28.895427 23042 log.cc:824] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:28.896157 23042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588: No bootstrap required, opened a new log
I20260925 14:53:28.897505 23042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.897648 23042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d07220ad6094547ade597171f1c8588, State: Initialized, Role: FOLLOWER
I20260925 14:53:28.897722 23042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.897971 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.898031 23042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:28.898041 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:28.898064 23016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 14:53:28.898187 23016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:28.899058 23016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.899611 23166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d07220ad6094547ade597171f1c8588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a94b6dd0b97d47b6a7632d800467ce93"
I20260925 14:53:28.899736 23166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [term 0 FOLLOWER]: Advancing to term 1
W20260925 14:53:28.900187 23223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:28.900236 23223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:28.900660 23166 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d07220ad6094547ade597171f1c8588 in term 1.
I20260925 14:53:28.900902 22992 leader_election.cc:302] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d07220ad6094547ade597171f1c8588, a94b6dd0b97d47b6a7632d800467ce93; no voters:
I20260925 14:53:28.901032 23212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:53:28.901181 23212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [term 1 LEADER]: Becoming Leader. State: Replica: 6d07220ad6094547ade597171f1c8588, State: Running, Role: LEADER
I20260925 14:53:28.901262 23212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.901659 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d07220ad6094547ade597171f1c8588. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.901744 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:28.901981 23016 leader_election.cc:288] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [CANDIDATE]: Term 1 election: Requested vote from peers 09bc78cc24a646a1b5bbfac4fcca79e9 (127.0.98.61:37193), a94b6dd0b97d47b6a7632d800467ce93 (127.0.98.60:40125)
I20260925 14:53:28.900629 22974 master_runner.cc:186] Error getting master registration for 127.0.98.61:37193: OK, instance_id { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" instance_seqno: 1790348008691275 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 14:53:28.902935 22974 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 14:53:28.903767 23227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:53:28.904428 23227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:53:28.906777 23166 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [term 1 FOLLOWER]: Refusing update from remote peer 6d07220ad6094547ade597171f1c8588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:28.907428 23225 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:28.908947 23198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d07220ad6094547ade597171f1c8588. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.909053 23198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:28.910290 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.910377 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: This master's current role is: LEADER
I20260925 14:53:28.910943 23227 catalog_manager.cc:1347] Generated new cluster ID: f8d6a79aff49456e821a874646d2c137
I20260925 14:53:28.910909 23198 sys_catalog.cc:455] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.910989 23227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:53:28.910988 23198 sys_catalog.cc:458] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:28.911275 23234 mvcc.cc:205] Tried to move back new op lower bound from 7333265444479524864 to 7333265444460437504. Current Snapshot: MvccSnapshot[applied={T|T < 7333265444479524864 or (T in {7333265444479524864})}]
I20260925 14:53:28.911546 23235 mvcc.cc:205] Tried to move back new op lower bound from 7333265444479524864 to 7333265444460437504. Current Snapshot: MvccSnapshot[applied={T|T < 7333265444479524864}]
I20260925 14:53:28.924985 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.903201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.0.98.62:46145)
0925 14:53:28.903244 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.903246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.914778 (+ 11532us) server_negotiation.cc:407] Connection header received
0925 14:53:28.914944 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.914961 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.914973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.915008 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.916241 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.916912 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.918421 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.918742 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.918792 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.921320 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.921322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.921322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.921330 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.923169 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.923728 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.923729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.923729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.923801 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.924206 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.924207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.924207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.924350 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.924424 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.924783 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.924862 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:53:28.926082 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.903594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33306 (local address 127.0.98.61:37193)
0925 14:53:28.903653 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.903655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.914762 (+ 11107us) server_negotiation.cc:407] Connection header received
0925 14:53:28.914892 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.914894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.914908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.914945 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.915832 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.916575 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.917905 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.918744 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.918791 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.922938 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.922940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.922940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.922948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.924818 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.925212 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.925214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.925214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.925275 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.925479 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.925480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.925480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.925609 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.925694 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.925903 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.925979 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:53:28.930202 23117 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } attempt: 3
I20260925 14:53:28.934844 23227 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:53:28.935387 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.906768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45026 (local address 127.0.98.60:40125)
0925 14:53:28.906810 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.906812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.917684 (+ 10872us) server_negotiation.cc:407] Connection header received
0925 14:53:28.917792 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.917793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.917805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.917839 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.920018 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.920752 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.923240 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.923541 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.923587 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.927651 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.927652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.927652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.927661 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.929192 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.934208 (+ 5016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.934209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.934210 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.934275 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.934784 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.934785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.934785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.934923 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.935007 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.935218 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.935297 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 14:53:28 dist-test-slave-z2s5 krb5kdc[22961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348008, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260925 14:53:28.936131 23227 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:53:28.940513 23120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.930628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40125 (local address 127.0.0.1:45030)
0925 14:53:28.930670 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 14:53:28.930680 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:53:28.930718 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:53:28.931047 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:53:28.931048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:53:28.931211 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:53:28.931252 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.931253 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.932701 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.932702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.933527 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:53:28.933529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.934273 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.934274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:53:28.934328 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.934711 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:53:28.934717 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:53:28.936320 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:53:28.939309 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.939310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.939311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.939521 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.939800 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:53:28.939801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:53:28.939801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:53:28.939865 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:53:28.940219 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:53:28.940219 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:53:28.940272 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:53:28.940320 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:53:28.940409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:53:28.941181 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:28.930494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.98.60:40125)
0925 14:53:28.930878 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.930880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.930887 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:53:28.930906 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.930906 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.930938 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.931331 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.931397 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.932560 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.933636 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:28.934156 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:28.934496 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:28.936467 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:53:28.936469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:53:28.936470 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:53:28.936478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:53:28.939186 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.939611 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.939612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.939612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.939712 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:53:28.939940 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:53:28.939941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:53:28.939941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:53:28.940127 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:53:28.940814 (+ 687us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:28.940878 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:53:28.940996 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":96,"threads_started":1}
I20260925 14:53:28.942889 23117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: Bootstrap starting.
I20260925 14:53:28.943615 23117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: Neither blocks nor log segments found. Creating new log.
I20260925 14:53:28.943905 23117 log.cc:824] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: Log is configured to *not* fsync() on all Append() calls
I20260925 14:53:28.944875 23117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: No bootstrap required, opened a new log
I20260925 14:53:28.947007 23227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588: Generated new TSK 0
I20260925 14:53:28.947014 23117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.947145 23117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bc78cc24a646a1b5bbfac4fcca79e9, State: Initialized, Role: FOLLOWER
I20260925 14:53:28.947219 23117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } }
I20260925 14:53:28.947289 23227 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:53:28.947548 23117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:53:28.947620 23091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6d07220ad6094547ade597171f1c8588" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9"
I20260925 14:53:28.947727 23091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:53:28.948542 23091 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d07220ad6094547ade597171f1c8588 in term 1.
I20260925 14:53:28.948858 23090 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [term 1 FOLLOWER]: Refusing update from remote peer 6d07220ad6094547ade597171f1c8588: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:53:28.950563 23225 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:53:28.952404 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.952541 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:28.952610 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d07220ad6094547ade597171f1c8588. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.952704 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:53:28.952775 23242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:28.952862 23242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:53:28.953259 23253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:53:28.953311 23253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:53:28.959055 23255 mvcc.cc:205] Tried to move back new op lower bound from 7333265444640776192 to 7333265444460437504. Current Snapshot: MvccSnapshot[applied={T|T < 7333265444479524864}]
W20260925 14:53:28.972491 23046 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.970783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42786 (local address 127.0.98.62:46145)
0925 14:53:28.970826 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.970828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.971813 (+ 985us) server_negotiation.cc:407] Connection header received
0925 14:53:28.971918 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.971919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.971931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.971962 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.972411 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42786: BlockingRecv error: recv got EOF from 127.0.0.1:42786 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:53:28.973289 23240 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.971209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.0.98.60:40125)
0925 14:53:28.971249 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.971251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.972784 (+ 1533us) server_negotiation.cc:407] Connection header received
0925 14:53:28.972878 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.972879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.972889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.972920 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.973228 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45040: BlockingRecv error: recv got EOF from 127.0.0.1:45040 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:53:28.974125 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.971544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.0.98.61:37193)
0925 14:53:28.971580 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.971581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.973567 (+ 1986us) server_negotiation.cc:407] Connection header received
0925 14:53:28.973689 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.973690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.973701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.973737 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.974059 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33314: BlockingRecv error: recv got EOF from 127.0.0.1:33314 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:53:28.976434 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.975734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33328 (local address 127.0.98.61:37193)
0925 14:53:28.975768 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.975770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.975950 (+ 180us) server_negotiation.cc:407] Connection header received
0925 14:53:28.976051 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.976052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.976069 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.976096 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.976393 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33328: BlockingRecv error: recv got EOF from 127.0.0.1:33328 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:53:28.979490 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.978769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33332 (local address 127.0.98.61:37193)
0925 14:53:28.978801 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.978803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.979017 (+ 214us) server_negotiation.cc:407] Connection header received
0925 14:53:28.979108 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.979109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.979118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.979145 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.979429 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33332: BlockingRecv error: recv got EOF from 127.0.0.1:33332 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 14:53:28.983426 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.982718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33346 (local address 127.0.98.61:37193)
0925 14:53:28.982749 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.982750 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.982962 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:53:28.983057 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.983058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.983067 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.983095 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.983383 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33346: BlockingRecv error: recv got EOF from 127.0.0.1:33346 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260925 14:53:28.989372 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.988622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33354 (local address 127.0.98.61:37193)
0925 14:53:28.988656 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.988657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.988895 (+ 238us) server_negotiation.cc:407] Connection header received
0925 14:53:28.988987 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.988988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.988999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.989028 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.989327 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33354: BlockingRecv error: recv got EOF from 127.0.0.1:33354 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 14:53:28.999692 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:28.998778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.0.98.61:37193)
0925 14:53:28.998821 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:28.998823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:28.999088 (+ 265us) server_negotiation.cc:407] Connection header received
0925 14:53:28.999196 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:28.999197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:28.999208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:28.999241 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:28.999642 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33368: BlockingRecv error: recv got EOF from 127.0.0.1:33368 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:53:29.018193 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:29.017169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33382 (local address 127.0.98.61:37193)
0925 14:53:29.017214 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.017216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.017510 (+ 294us) server_negotiation.cc:407] Connection header received
0925 14:53:29.017639 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.017640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.017653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.017686 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.018133 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33382: BlockingRecv error: recv got EOF from 127.0.0.1:33382 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:53:29.052738 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:29.051778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33388 (local address 127.0.98.61:37193)
0925 14:53:29.051826 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.051828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.052112 (+ 284us) server_negotiation.cc:407] Connection header received
0925 14:53:29.052220 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.052221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.052233 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.052265 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.052675 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33388: BlockingRecv error: recv got EOF from 127.0.0.1:33388 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 14:53:29.118877 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:29.118182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33398 (local address 127.0.98.61:37193)
0925 14:53:29.118236 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.118238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.118372 (+ 134us) server_negotiation.cc:407] Connection header received
0925 14:53:29.118418 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.118419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.118431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.118487 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.118818 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33398: BlockingRecv error: recv got EOF from 127.0.0.1:33398 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260925 14:53:29.248656 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:29.248013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.0.98.61:37193)
0925 14:53:29.248073 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.248075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.248177 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:53:29.248226 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.248227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.248239 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.248303 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.248609 (+ 306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33408: BlockingRecv error: recv got EOF from 127.0.0.1:33408 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:53:29.326893 23225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6d07220ad6094547ade597171f1c8588" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d07220ad6094547ade597171f1c8588" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46145 } } peers { permanent_uuid: "09bc78cc24a646a1b5bbfac4fcca79e9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 37193 } } peers { permanent_uuid: "a94b6dd0b97d47b6a7632d800467ce93" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40125 } } }
I20260925 14:53:29.327054 23225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d07220ad6094547ade597171f1c8588 [sys.catalog]: This master's current role is: LEADER
W20260925 14:53:29.506554 23122 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:53:29.505870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.0.98.61:37193)
0925 14:53:29.505937 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:29.505939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:29.505950 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:53:29.506014 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:29.506015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:29.506027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:29.506102 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:53:29.506466 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33414: BlockingRecv error: recv got EOF from 127.0.0.1:33414 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:53:29.884763 23209 catalog_manager.cc:1259] Loaded cluster ID: f8d6a79aff49456e821a874646d2c137
I20260925 14:53:29.884866 23209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: loading cluster ID for follower catalog manager: success
I20260925 14:53:29.885747 23209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: acquiring CA information for follower catalog manager: success
I20260925 14:53:29.886068 23209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a94b6dd0b97d47b6a7632d800467ce93: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:53:29.953724 23253 catalog_manager.cc:1259] Loaded cluster ID: f8d6a79aff49456e821a874646d2c137
I20260925 14:53:29.953828 23253 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: loading cluster ID for follower catalog manager: success
I20260925 14:53:29.954655 23253 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: acquiring CA information for follower catalog manager: success
I20260925 14:53:29.955016 23253 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 09bc78cc24a646a1b5bbfac4fcca79e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:53:30.022543 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.019788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.0.98.61:37193)
0925 14:53:30.019919 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.019921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.019933 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:53:30.020032 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.020033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.020101 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.020193 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:30.020354 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.020875 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.021686 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.021895 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.021972 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:30.022170 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:30.022278 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:30.022369 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.022393 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.022444 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":59,"threads_started":1}
I20260925 14:53:30.027108 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.024517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45046 (local address 127.0.98.60:40125)
0925 14:53:30.024646 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.024649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.024659 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:53:30.024743 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.024744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.024791 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.024882 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:30.025065 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.025563 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.026322 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.026584 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.026649 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:30.026781 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:30.026882 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:30.026937 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.026954 (+ 17us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.026970 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":56,"threads_started":1}
I20260925 14:53:30.031935 23333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.028856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45054 (local address 127.0.98.60:40125)
0925 14:53:30.028893 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.028895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.029071 (+ 176us) server_negotiation.cc:407] Connection header received
0925 14:53:30.029219 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.029220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.029232 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.029282 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:30.029386 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.030041 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.031166 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.031449 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.031507 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:30.031685 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:30.031715 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:30.031768 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.031823 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.031845 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:53:30.032015 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.028787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.0.98.61:37193)
0925 14:53:30.028830 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.028839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.029064 (+ 225us) server_negotiation.cc:407] Connection header received
0925 14:53:30.029130 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.029131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.029143 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.029222 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:30.029676 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.030185 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.031259 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.031513 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.031582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:30.031733 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:30.031769 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:30.031817 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.031907 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.031934 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:53:30.032302 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:53:30.028791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42802 (local address 127.0.98.62:46145)
0925 14:53:30.028955 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 14:53:30.028957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:53:30.029126 (+ 169us) server_negotiation.cc:407] Connection header received
0925 14:53:30.029351 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:53:30.029352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:53:30.029389 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:53:30.029494 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:53:30.029639 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.030305 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.031070 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:53:30.031390 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:53:30.031459 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:53:30.031675 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:53:30.031789 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:53:30.031846 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:53:30.032020 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:53:30.032214 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260925 14:53:30.033763 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 22974
I20260925 14:53:30.039162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 23049
I20260925 14:53:30.044018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskM77lMC/build/release/bin/kudu with pid 23124
2026-09-25T14:53:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2809 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5616 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271005 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST